Display plots from different URIs

[mfiaudrin-epsyl]

This PR aims to use interpolation in order to allow users to display several plots from diffent URIs in a same graph.

[read-the-docs-community]

Documentation build overview

📚 imas-ibex | 🛠️ Build #32795103 | 📁 Comparing 5ed9ba8 against latest (0c56cfe)

  🔍 Preview build  

48 files changed · ± 48 modified

± Modified

• index.html
• users_manual/features.html
• users_manual/how_to_launch.html
• developers_manual/backend_development/adding_new_data_source.html
• developers_manual/backend_development/adding_new_downsampling_method.html
• developers_manual/backend_development/api-hidden.html
• developers_manual/backend_development/api.html
• developers_manual/backend_development/backend_development.html
• developers_manual/backend_development/backend_development_introduction.html
• developers_manual/backend_development/benchmarking.html
• and 38 more...

[mfiaudrin-epsyl]

Hi @jwasikpsnc,

I am implementing the feature to display several plots from different URIs using your interpolation functionality.

I have encountered a bug with the interpolation calling data/plot_data (Code 500). There are my params:

• uri: imas:hdf5?path=/home/ITER/tribolp/public/imasdb/ITER/3/135011/98#pf_active:0/coil[:]/current/data
• interpolate_over: imas:hdf5?path=/work/imas/shared/imasdb/ITER/3/135011/7/#pf_active:0/coil[:]/current/data

And there is the full route:
http://127.0.0.1:52879/data/plot_data?uri=imas%3Ahdf5%3Fpath%3D%2Fhome%2FITER%2Ftribolp%2Fpublic%2Fimasdb%2FITER%2F3%2F135011%2F98%23pf_active%3A0%2Fcoil%5B%3A%5D%2Fcurrent%2Fdata&interpolate_over=imas%3Ahdf5%3Fpath%3D%2Fwork%2Fimas%2Fshared%2Fimasdb%2FITER%2F3%2F135011%2F7%2F%23pf_active%3A0%2Fcoil%5B%3A%5D%2Fcurrent%2Fdata

I share you a screenshot of the error, it looks that string type is not handle by the interpolation:

Can you please do some investigations and fix it?
Thanks !

[mfiaudrin-epsyl]

Hi @jwasikpsnc,

Is it possible to also have the interpolate_over parameter in the data/field_value route ?

[jwasikpsnc]

Not at the moment. What do you need it for?

[mfiaudrin-epsyl]

Ok, at the moment I can use data/plot_data since it's not blocking.

I would like to call a first time the interpolation with data/plot_data to have the new selected plot to include in the graph with the new common coordinates.

I will then have to update the other plot(s) of the graph to also interpolate them. I would like to do it by calling data/field_value because I just need to have the data of the plot since I already have the coordinates.

I think it would perform a little better since less data will be send. But we can discuss of that later if it's a lot of works for an imperceptible improves.

[jwasikpsnc]

data/field_value handles only data (without coordinates) extracted from IMAS database.
In order to interpolate data i need original coordinates that I already have in data/plot_data endpoint logic.
Getting access to coordinates in data/field_value would need extensive development and would make this endpoint too similar to data/plot_data, thus it would be better to use plot_data in this case.

After all, AFAIR you use data/field_falue to extract error bands.
Before, we didn't had to worry about coordinates, but now we have to.
In this case I think it would be reasonable to use plot_data to get error bands using logic simillar to this:

if not interpolate:
    error_bands = backend.field_value(...)
else:
    error_bands = backend.plot_data(interpolate_over=...)

[mfiaudrin-epsyl]

Hi @jwasikpsnc,
The interpolation isn't working as expected when I try to get error bands.

Here is an example where I want to get an ip_error_upper data with an interpolation based on A/ip, B/ip calling /data/plot_data:

• uri : imas:hdf5?user=public;pulse=100002;run=1;database=iterdb;version=3#equilibrium:0/time_slice[:]/global_quantities/ip_error_upper
• interpolate_over:
  • imas:hdf5?user=public;pulse=100002;run=1;database=iterdb;version=3#equilibrium:0/time_slice[:]/global_quantities/ip (this one is probably useless because A/ip_error_upper and A/ip should have same coordinates)
  • imas:hdf5?path=/home/ITER/tribolp/public/imasdb/ITER/3/135011/98#equilibrium:0/time_slice[:]/global_quantities/ip

The same problem occurs for ip_error_lower.

The error received is (466 Code): Error: IDS name and node path should be the same for source and target URI when interpolating data.

It looks that the interpolation is only working for same paths so error_upper and error_lower aren't interpolables.

Can you do some investigations and fix it ?

[jwasikpsnc]

Hi @mfiaudrin-epsyl
This is intended behavior.
You should put URI1/time_slice[:]/global_quantities/ip_error_upper in uri argument
and URI2/time_slice[:]/global_quantities/ip_error_upper in interpolate_over.
Both URIs must point to the same path.
Do not repeat the same URI in both arguments.
Please let me know if this answer finds you well.

[mfiaudrin-epsyl]

@jwasikpsnc Thank you for your feedback, it should works when all URIs have error band data.
Unfortunately in this case there is no error band data in URI B so I can't interpolate the error bands from URI A.

Do you think we can Add a BE rule to allow to interpolate with a different path only when the difference is at the end with "_error_upper" or "_error_lower" since coordinates should be the same between the plotted data and his error_upper / error_lower ?

[jwasikpsnc]

Ok, now i understand
I will open new issue

[mfiaudrin-epsyl]

Hello @olivhoenen,
I have implemented the E2E test to ensure that coordinates between 2 plots from different URIs have a common one.
I set the PR to ready so that you can test it and approves it.

To remind, this feature allow the user to plot several plots from different URIs in a same graph and compare them in a scientific way thanks to the union of coordinates to have a common ones.
Feel free to test with different data. There is the ones I used for this implementation:

• URI 1 : https://zenodo.org/records/17062700/files/iter_scenario_53298_seq1_DD3.nc
• URI 2 : https://zenodo.org/records/17062700/files/iter_disruption_113112_1.nc
• Path : #equilibrium/time_slice[0]/profiles_1d/psi

NB: This feature include interpolation in plus of combination of coordinates. It has been asked by ITER to remove it. It should be done from BE in an other branch. No changes are expected from FE.

[mfiaudrin-epsyl]

Hello @olivhoenen,
I've juste rebase this feature with develop to includes updates from PSNC (#60).
You can test it, it works as I expected

[olivhoenen]

Thanks @mfiaudrin-epsyl @jwasikpsnc . As discussed, here are the remaining issues:

• plots of a given signal shall not be discontinued (https://stackoverflow.com/questions/71900162/plotly-how-to-plot-multiple-lines-with-different-x-arrays-on-the-same-y-axis).
• while case with equilibrium/time_slice/profiles_1d/psi works as expected as indicated in this PR, other cases seem not to work, e.g. equilibrium/time_slice/profiles_1d/q or the simpler summary/global_quantities/ip/value. See attached configuration to reproduce compare-metisIbexState.json
• by default y-axis title shall not be psi, ... [Wb] but only psi [Wb] if only psi quantities are plotted on this axis (and I created issue Allow editing the x/y-axis titles in plot customization #71 on the possibility to edit these titles)

[mfiaudrin-epsyl]

@olivhoenen @jwasikpsnc,
FYI I have merged from develop to get #59.

I have done some tests from my side and it looks that we can plot from different URIs with error bands.

However, there is a strange behavior where error bands aren't the same if we compare 2 ways to plot the data:

• Add plot 1, disable error_bands, add plot 2, activate error_bands (OK)
• Add plot 1, keep activated error_bands and add plot 2 (KO)

I will do some investiong in FE side and will let you know if it cames from me.

[mfiaudrin-epsyl]

Hello @olivhoenen,
I just implemented your feedbacks about :

1. Handling error bands with combined coordinates (FYI @jwasikpsnc the unexpected behavior I mentionned was in FE since I was required your update to handle properly error bands with combined coordinates)
2. Updating the rule for y titles to prevent ‘…’ from appearing when titles are identical
3. Maintaining a continuous curve when using combined coordinates

About allowing users to update x and y titles, it will be done in a futur PR.

This PR is ready to be reviewed.

[olivhoenen]

Thanks @mfiaudrin-epsyl. It looks like you tackled all the points from #55 (comment) except the central point, which is an important one as it stops us using the feature in many cases.

I've added a new config compare-testIbexState.json this time trying to plot core_profiles/profiles_1d/electrons/temperature from 2 or 3 URIs. This one shall be even simpler as the data for each URI seems to have the same shape. Plotting them separately looks all ok, but when one tries to add a second URI to one of these plots we get the following error:

==========> Endpoint 'node_info' executed in 0.0165 seconds
INFO:     127.0.0.1:33316 - "GET /ids_info/node_info?uri=imas%3Ahdf5%3Fpath%3D%2Fwork%2Fimas%2Fshared%2Fimasdb%2FITER%2F3%2F100120%2F1%23core_profiles%3A0%2Fprofiles_1d%2Felectrons&show_error_bars=false HTTP/1.1" 200 OK
15:20:27 ERROR    Traceback (most recent call last):                                                                                                                                                                
  File "/home/ITER/hoeneno/.config/ibex/ibex_venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app                                                                            
    await app(scope, receive, sender)                                                                                                                                                                               
  File "/home/ITER/hoeneno/.config/ibex/ibex_venv/lib/python3.11/site-packages/starlette/routing.py", line 73, in app                                                                                               
    response = await f(request)                                                                                                                                                                                     
               ^^^^^^^^^^^^^^^^                                                                                                                                                                                     
  File "/home/ITER/hoeneno/.config/ibex/ibex_venv/lib/python3.11/site-packages/fastapi/routing.py", line 301, in app                                                                                                
    raw_response = await run_endpoint_function(                                                                                                                                                                     
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                     
  File "/home/ITER/hoeneno/.config/ibex/ibex_venv/lib/python3.11/site-packages/fastapi/routing.py", line 212, in run_endpoint_function                                                                              
    return await dependant.call(**values)                                                                                                                                                                           
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                           
  File "/home/ITER/hoeneno/GIT/IBEX/backend/ibex/core/ibex_service.py", line 19, in wrapper                                                                                                                         
    response = func(*args, **kwargs)                                                                                                                                                                                
               ^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                
  File "/home/ITER/hoeneno/GIT/IBEX/backend/ibex/endpoints/data.py", line 125, in plot_data                                                                                                                         
    ibex_service.get_plot_data(                                                                                                                                                                                     
  File "/home/ITER/hoeneno/GIT/IBEX/backend/ibex/core/ibex_service.py", line 125, in get_plot_data                                                                                                                  
    return data_source.get_plot_data(                                                                                                                                                                               
           ^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                               
  File "/home/ITER/hoeneno/GIT/IBEX/backend/ibex/data_source/imas_python_source.py", line 854, in get_plot_data                                                                                                     
    data_to_be_returned = resample_data_without_interpolation(                                                                                                                                                      
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                      
  File "/home/ITER/hoeneno/GIT/IBEX/backend/ibex/data_source/imas_python_source_utils.py", line 193, in resample_data_without_interpolation                                                                         
    result = data                                                                                                                                                                                                   
    ~~~~~~^^^^^^^^^^^^^                                                                                                                                                                                             
ValueError: shape mismatch: value array of shape (175,21) could not be broadcast to indexing result of shape (175,3327)                                                                                             
 @exception_handlers.py:16                                                                                                                                                                                          
INFO:     127.0.0.1:33316 - "GET /data/plot_data?uri=imas%3Ahdf5%3Fpath%3D%2Fwork%2Fimas%2Fshared%2Fimasdb%2FITER%2F3%2F100120%2F1%23core_profiles%3A0%2Fprofiles_1d%5B%3A%5D%2Felectrons%2Ftemperature&interpolate_over=imas%3Ahdf5%3Fpath%3D%2Fwork%2Fimas%2Fshared%2Fimasdb%2FITER%2F3%2F100020%2F1%23core_profiles%3A0%2Fprofiles_1d%5B%3A%5D%2Felectrons%2Ftemperature HTTP/1.1" 500 Internal Server Error
==========> Endpoint 'version' executed in 0.0000 seconds

(175,21) seems to be the valid shape in both cases, no idea where the (175,3327) comes from in the backend, @jwasikpsnc ?