Faster tests

[tgrandje]

#299 and #296

[tgrandje]

In the present state, that PR should'nt be merged until we state on #303 : it uses a recent geopandas functionality which is not available in the latest py3.9-compatible geopandas release.

[tgrandje]

OK, this is good for review.

I dumped most of the tree decision managing department's retrieval as it was not easy to maintain and could somehow be quite slow. It's been simplified to 3 cases:

• either we're working on the department's dataset (same as before)
• or we're on a dataset where INSEE (now) displays insee dpt codes (that's the case for EPCIs for instance) ;
• or in any other case, it's fast enough to use a sjoin (assuming we already stored a preprocessed departments' geodataset).

During my manual tests, I've mostly seen awkward results for "collectivités territoriales" (Saint-Martin, Saint-Barthélémy) or some other exotic cases : for instance arrondissements municipaux datasets, cantons, where there are holes in french territory or when you choose to zoom in on Paris' inner suburbs and your current dataset has geometries maching Paris's external suburbs. I don't think that was ever perfectly rendered in pynsee...

[tfardet]

shouldn't this occur only in the if? (otherwise there is no overlap so I don't see why we should change the correct polygons

[tgrandje]

No : if a geometry fully covers a city department, it will sjoin on a city and it's neighbours. The buffer of -1meter is gonna fix this in the overwhelming majority of cases.

EDIT : I meant a department and not a city, obviously... 😳

[tfardet]

sorry, I'm not getting this, could you maybe draw it on a piece of paper and take a photo? (or we can schedule a meeting to discuss it in real time if it's easier, in which case you can just email me)
To prepare for that discussion, even though I don't get the issue, I'm wondering why you chose not to test for small fractions of the original areas instead of modifying the original geometries.

[tgrandje]

[tgrandje]

https://excalidraw.com/#json=sUTRvCvLrtHnFMbbEKmxF,O1CsByZYwfowqnS-xUqb7A

[tgrandje]

Is this clearer?

EDIT
The negative buffer helps avoiding case 1A. Problems might occur on case 2B, but that seems too exotic a case to be handled by pynsee (I think this will concern only points in a 2meters-large bandwidth around DROM territories).

[tgrandje]

I'm wondering why you chose not to test for small fractions of the original areas instead of modifying the original geometries.

I'm not sure I understood what you meant either. Let's discuss this in our matrix chanel?

[tfardet]

I don't think the buffer part is relevant here anymore, is it?

[tgrandje]

This is still useful for case 1A (see above).

[tfardet]

adapt to "may" if you indeed moved the buffer into the if

[tgrandje]

I fxed the comment (not a 10 meter range, but 1 meter)

[tgrandje]

Note for memory's sake: we also could detect multipoint datasets and link it to parent datasets. For instance chef_lieu_d_epci has a lien_vers_l_epci fields which links to cleabs in the epci dataset. But:

• that seems overly complex to maintain (it's a rare use case I think) while sjoin are fasts on a point vs. polygon match,
• and users may use the LATEST dataset vintage and there is no certitude that LATEST:chef_lieu_d_epci and LATEST:epci are of the same vintage.