Merging many experiments, tedious?

Share icon

Sun, 04/17/2016 - 02:13 pm

I have many, many data sets. Each data set contains 5 different types of data. I have exported each graph of each type of data into a packed experiment file with the date of the experiment and some other parameters.

Now I'm at the point where I want to merge each of these packed experiment files into one LARGE experiment for each type of data. I have to merge ~50 packed experiment files for each type of data. So I will end up with 5 different packed experiments, one for each type of data. This way I can compare all of the same data to itself. Yes, that takes the data out of context but my data are very consistant, and now I'm just looking for the best looking data to create figures for a manuscript.

Each packed experiment file contains only one graph, with corresponding data, so I'm not expecting data conflicts or naming conflicts, however I would like a quicker way than just option clicking open experiment 250 times...

Any way to do this more quickly?

EDIT: And, for the love of god, is there a way to disable to freaking merge experiments warning?

EDIT2: Also, the option clicking of "Open experiments" is inconsistent. Sometimes it works, other times it just considers it a normal "open experiments." I do use a wireless keyboard, but it has brand new batteries. I don't think it's a problem with the keyboard, but I'm not going to test it further. It's just annoying. It'll often happen 2-3 times in a row where I'll get the "Do you want to save changes" dialog instead of the "Merging Experiments" warning.
Hi,

Maybe another approach, can you save the existing experiments as unpacked? That will expose the individual waves as files You can then script something to load the waves from the appropriate directories.

I would create a master list of the experiments and then run through and re save them to a new master location as unpacked (just to be safe and not mess up the originals). Once I have the master list, I can create a load procedure that picks up the needed files from the directories.

Perhaps a way to start.

Andy

April 17, 2016 at 06:41 pm - Permalink

LoadData is the operation you should use to load data from a packed experiment file (.pxp). If you do not have naming conflicts, loading the data should be straight forward.

April 17, 2016 at 08:31 pm - Permalink

hegedus wrote:
Hi,

Maybe another approach, can you save the existing experiments as unpacked? That will expose the individual waves as files You can then script something to load the waves from the appropriate directories.

I would create a master list of the experiments and then run through and re save them to a new master location as unpacked (just to be safe and not mess up the originals). Once I have the master list, I can create a load procedure that picks up the needed files from the directories.

Perhaps a way to start.

Andy


I may look more into this, however I have the same problem here as I have with the response below yours.

I'm merging the experiments because each of the graphs contains up to 20 different x-y wave combinations (20 individual traces). That's 40 1D waves. Multiply that by 50 or so graphs and you quickly get into the realm of "not wanting to have to create all of the graphs again." While I certainly COULD, as I have the procedures to create graphs quickly, it's not the point. I'm a human. If I had to exactly recreate 250 different graphs, I'm bound to make a mistake here and there.

Besides, I don't think my procedures would work because they're not written to take into account different data folders.

If I wanted to simply load all of the data I would just use one of my many multiload programs, however that's not what I'm trying to accomplish.

I did... eventually... merge all of the experiments yesterday. It took about 4 hours of mindless repetitive clicking. I would simply like a way to make that process much quicker.

I mean, all they have to do is allow you to merge multiple experiments at once. Shift click the merge experiment open file dialog... that's all. (Of course, I know there are probably problems with that.)

April 18, 2016 at 07:12 am - Permalink

Question for reepingk:

Are you comfortable in scripting? For myself if I have to do anything more than a couple of times (and that number is getting smaller with practice), I will write a script.

I find if I had to do it once, odds are pretty good I will need to do it again and developing a script not only saves me some time, but also produces better results because the chance of random errors goes down. The chance of systematic errors goes down, once I debug the script.

Andy

April 18, 2016 at 07:48 am - Permalink

You can merge experiments using Execute/P with MERGEEXPERIMENT.

See:
DisplayHelpTopic "Operation Queue"

Quote:
Maybe another approach, can you save the existing experiments as unpacked? That will expose the individual waves as files You can then script something to load the waves from the appropriate directories.


You can load data without saving as unpacked using the LoadData operation.

April 18, 2016 at 08:02 am - Permalink

hegedus wrote:
Question for reepingk:

Are you comfortable in scripting? For myself if I have to do anything more than a couple of times (and that number is getting smaller with practice), I will write a script.

I find if I had to do it once, odds are pretty good I will need to do it again and developing a script not only saves me some time, but also produces better results because the chance of random errors goes down. The chance of systematic errors goes down, once I debug the script.

Andy


Of course. I'll probably write a script to do it after I finish writing this manuscript. In all honesty I didn't even look up if there was a command for "Merge Experiments." (As it would have taken longer to perfect the script than it would have to just do it manually, and it had to be done yesterday.)

April 18, 2016 at 09:35 am - Permalink