[Flightgear-scenery] Testing for point/line in polygon problem

[Adrian Musceac]

Hi guys,

I know there's not much activity on this list, but I have a question directed
at the terragear developers especially.

I have a list of coordinates, namely elevation points, and a list of polygons,
obtained by ogr-decode (lots of polygons, obtained through Corine).

Now, I've written my own tool which uses gdal/ogr with GEOS bindings, in order
to perform the OGRGeometry::Within() test for each point. (Actually I'm using
a very small line because points tend to give me an exception).

So far so good, except for the fact that I have about 900 tables each with
12000 + coordinate points. The Within test performed by GEOS takes too long.
I've calculated that I would need 40 days to fill my database which is
unacceptable.

I wonder if anybody has a better solution to my problem rather than using the
point-in-poly routine of GEOS. I've not tried shapefiles, because ogr-decode
gives me exactly the name/type of terrain used by Flightgear.

I'm considering writing a program using an OSG intersect visitor on each small
btg tile, but I don't know whether that would be any faster at all.

Thanks for any answers.

Cheers,
Adrian

[HB-GRAL]

Hi Adrian
Maybe you can try postgis (which uses GEOS too) and compare performance
with your tool/own bindings?
-Yves

Hi Yves,
It's actually not the database format but rather gdal/ogr with GEOS that's
slow. I'll try more research in postgis, but aren't they using as a backend
OGR/GDAL too?
Cheers,
Adrian

Hi Adrian

Maybe this link could contain some useful information for you, I don’t
know ...
http://trac.osgeo.org/postgis/wiki/WKTRasterTutorial01

As far as I understand you're working with sqlite. I don’t know if this
is the best dbms for what you’re doing (problems with indexing, filelock
etc.). Here is a postgres alternative which extends sqlite core for GIS:
https://www.gaia-gis.it/fossil/libspatialite/index

-Yves

[Adrian Musceac]

Hi Adrian
Maybe this link could contain some useful information for you, I don’t
know ...
http://trac.osgeo.org/postgis/wiki/WKTRasterTutorial01
As far as I understand you're working with sqlite. I don’t know if this
is the best dbms for what you’re doing (problems with indexing, filelock
https://www.gaia-gis.it/fossil/libspatialite/index
-Yves

Hi Yves,
I have to work with sqlite as it is an embeddable database (like you can take
the file and use it anywhere). For my purposes it's the only solution.
But the bottleneck is not in sqlite anymore.
Meanwhile I've identified a bug which made my code incredibly slow.
Right now I'm looking at 10 full days to fill a 10x10 degree elevation
database with material info. I will try to optimize even further, I believe I
can. I'm using output from ogr-decode, srtmchop and terrafit. I will try to
keep the wiki article updated too.

I tried using GEOS directly but it segfaults quite often with latest version,
so I abandoned it. Right now I'm using GDAL/OGR with GEOS bindings.

With this, terrain sampling 100-1000 km away will be possible, as long as one
keeps the number of sampled points reasonable. Size of the database remains a
problem, as well as time to get it.

Cheers,
Adrian

[Peter Sadrozinski]

Possibly more difficult to set up, but I'm adding more and more CGAL
routines to terragear.

I just commited a find points within bounding box which improved my lookup
times by 20%.

To find points within, what you really want is to create a CGAL
arrangement, then you can query a point and it should be really quick.

See
http://www.cgal.org/Manual/3.3/doc_html/cgal_manual/Arrangement_2/Chapter_main.html

Pete

Hi Adrian
Maybe this link could contain some useful information for you, I don’t
know ...
http://trac.osgeo.org/postgis/wiki/WKTRasterTutorial01
As far as I understand you're working with sqlite. I don’t know if this
is the best dbms for what you’re doing (problems with indexing, filelock
https://www.gaia-gis.it/fossil/libspatialite/index
-Yves

Hi Yves,
I have to work with sqlite as it is an embeddable database (like you can take
the file and use it anywhere). For my purposes it's the only solution.
But the bottleneck is not in sqlite anymore.
Meanwhile I've identified a bug which made my code incredibly slow.
Right now I'm looking at 10 full days to fill a 10x10 degree elevation
database with material info. I will try to optimize even further, I believe I
can. I'm using output from ogr-decode, srtmchop and terrafit. I will try to
keep the wiki article updated too.
I tried using GEOS directly but it segfaults quite often with latest version,
so I abandoned it. Right now I'm using GDAL/OGR with GEOS bindings.
With this, terrain sampling 100-1000 km away will be possible, as long as one
keeps the number of sampled points reasonable. Size of the database remains a
problem, as well as time to get it.
Cheers,
Adrian
------------------------------------------------------------------------------
LogMeIn Rescue: Anywhere, Anytime Remote support for IT. Free Trial
Remotely access PCs and mobile devices and provide instant support
Improve your efficiency, and focus on delivering more value-add services
Discover what IT Professionals Know. Rescue delivers
http://p.sf.net/sfu/logmein_12329d2d
_______________________________________________
Flightgear-scenery mailing list
https://lists.sourceforge.net/lists/listinfo/flightgear-scenery

[Adrian Musceac]

Post by Peter Sadrozinski
Possibly more difficult to set up, but I'm adding more and more CGAL
routines to terragear.
I just commited a find points within bounding box which improved my lookup
times by 20%.
To find points within, what you really want is to create a CGAL
arrangement, then you can query a point and it should be really quick.
See
http://www.cgal.org/Manual/3.3/doc_html/cgal_manual/Arrangement_2/Chapter_m
ain.html
Pete

Hi Pete,
I really hope my toolchain will work on the next issue of terragear :)
Do we still have terrafit which outputs .fit.gz, and ogr-decode does function
the same way? Right now I'm on the right path, but it's bumpy.
I think there's a real need for a LOD system, or another system which would
allow loading terrain as far as 300 km away, perhaps without adding at load
time random elements, lights, and stuff. Unfortunately I'm not very familiar
with OSG database pagers and the underlyings in simgear.

I'm off to research CGAL now.

Cheers,
Adrian

[Peter Sadrozinski]

Actually, we are in the process of modifying quite a few of these
intermediate files. But the changes should make it easier for you long
term.

Hgtfit should now be saving binary data instead of text to the .gz file

I'm in the 'thinking' stage for ogr-decode. I'll probably change the
output polygon format quite a bit. But the new terragear polygon class
(TGPolygon is being refactored to tgPolygon) has the capability to load and
save themselves to a gz stream. You will most likely be able to do this.

I'm working on this now. Should be ready in a couple weeks.

Ogr-decode appears io bound on my computer. I'm going to try to limit the
shear number of files created by ogr-decode.

Pete

Post by Peter Sadrozinski

Post by Peter Sadrozinski
Possibly more difficult to set up, but I'm adding more and more CGAL
routines to terragear.
I just commited a find points within bounding box which improved my

lookup

Post by Peter Sadrozinski
times by 20%.
To find points within, what you really want is to create a CGAL
arrangement, then you can query a point and it should be really quick.
See

http://www.cgal.org/Manual/3.3/doc_html/cgal_manual/Arrangement_2/Chapter_m

Post by Peter Sadrozinski
ain.html
Pete

Hi Pete,
I really hope my toolchain will work on the next issue of terragear :)
Do we still have terrafit which outputs .fit.gz, and ogr-decode does function
the same way? Right now I'm on the right path, but it's bumpy.
I think there's a real need for a LOD system, or another system which would
allow loading terrain as far as 300 km away, perhaps without adding at load
time random elements, lights, and stuff. Unfortunately I'm not very familiar
with OSG database pagers and the underlyings in simgear.
I'm off to research CGAL now.
Cheers,
Adrian
------------------------------------------------------------------------------
LogMeIn Rescue: Anywhere, Anytime Remote support for IT. Free Trial
Remotely access PCs and mobile devices and provide instant support
Improve your efficiency, and focus on delivering more value-add services
Discover what IT Professionals Know. Rescue delivers
http://p.sf.net/sfu/logmein_12329d2d
_______________________________________________
Flightgear-scenery mailing list
https://lists.sourceforge.net/lists/listinfo/flightgear-scenery

[Christian Schmitt]

Post by Adrian Musceac
I really hope my toolchain will work on the next issue of terragear :)
Do we still have terrafit which outputs .fit.gz, and ogr-decode does
function the same way? Right now I'm on the right path, but it's bumpy.
I think there's a real need for a LOD system, or another system which
would allow loading terrain as far as 300 km away, perhaps without adding
at load time random elements, lights, and stuff. Unfortunately I'm not
very familiar with OSG database pagers and the underlyings in simgear.

Hi Adrian,

generally, yes a LOD system would be nice, but it would need considerable
changes on the btg format or a switch to something new.
I'm currently investigating how to make elevation calculation for the
terrain easier and more versatile. I want to use gdalchop for general
processing as gdal reads almost all kinds of raster formats. So we would no
longer have to rely on hgtchop, srtmchop and demchop (yes, it's mad).
Also, depending on what is possible, i would like to get rid of terrafit. We
have great SRTM data, reduce the amount of points and then have to
interpolate like mad when we need the elevation of a certain point. GDAL has
interpolation routines included (beyond linear interpolation) and I'd like
to make use of that. I noticed yesterday, that there is already
gdallocationinfo which reads the elevation on the fly for a given lat/lon.
This might be worth investigating.

Cheers
Chris

[Adrian Musceac]

Post by Christian Schmitt
Hi Adrian,
generally, yes a LOD system would be nice, but it would need considerable
changes on the btg format or a switch to something new.
I'm currently investigating how to make elevation calculation for the
terrain easier and more versatile. I want to use gdalchop for general
processing as gdal reads almost all kinds of raster formats. So we would no
longer have to rely on hgtchop, srtmchop and demchop (yes, it's mad).
Also, depending on what is possible, i would like to get rid of terrafit.
We have great SRTM data, reduce the amount of points and then have to
interpolate like mad when we need the elevation of a certain point. GDAL
has interpolation routines included (beyond linear interpolation) and I'd
like to make use of that. I noticed yesterday, that there is already
gdallocationinfo which reads the elevation on the fly for a given lat/lon.
This might be worth investigating.
Cheers
Chris

Hi Chris,

You are correct about SRTM and GDAL, but I'd like to add that my greatest
problem now is not reading the elevation (this is already done, albeit
innacurately), but reading and placing the material information in the same
table as the elevaton info. And this I can currently do only by using polygon
output from ogr-decode. Also, I'm relying on text files generated by terrafit,
because I have a simple python script to get that into the database. I'll see
how to adapt in the future.
The wonders of using sqlite is that you can attach and detach on the fly
databases to the main connection, and if you have unique tables, the amount of
code needed is really small to handle transition from one area to another.
I already have elevation sampling working in Flightgear with 10x10 degree
tiles (for whatever distance that allows), the very nasty trick is obtaining
that material information for each point and I'm currently working on a fast
method to get that info.
Right now I take polygons as generated by ogr-decode, and apply the GEOS
routine Within test for each elevation point. Considering 12,000,000 elevation
points for a 10x10 tile, and lots of polygons generated by ogr-decode, it's
literally taking days or weeks to complete this.

I will dedicate some time in the future to think about terrain LOD without
significant changes to the BTG format. The current system is definetly
inferior to most every other simulator out there.

Cheers,
Adrian

[HB-GRAL]

Post by Christian Schmitt

Post by Adrian Musceac
I really hope my toolchain will work on the next issue of terragear :)
Do we still have terrafit which outputs .fit.gz, and ogr-decode does
function the same way? Right now I'm on the right path, but it's bumpy.
I think there's a real need for a LOD system, or another system which
would allow loading terrain as far as 300 km away, perhaps without adding
at load time random elements, lights, and stuff. Unfortunately I'm not
very familiar with OSG database pagers and the underlyings in simgear.

Hi Adrian,
generally, yes a LOD system would be nice, but it would need considerable
changes on the btg format or a switch to something new.
I'm currently investigating how to make elevation calculation for the
terrain easier and more versatile. I want to use gdalchop for general
processing as gdal reads almost all kinds of raster formats. So we would no
longer have to rely on hgtchop, srtmchop and demchop (yes, it's mad).

Hi Christian

Nice to hear that there are some plans (?) with extending gdalchop. As I
mentioned earlier I’m preparing 5x5 degree tiles of all the SRTM data in
geotiff format to use with terragear (just in case someone wants to give
it a try once, I use it for my reliefs and this 5x5 tiles are a side
effect of processing this data). A 5x5 degree geotiff uncompressed will
have around 72 MB.

When it’s useful and gdalchop can handle this I can prepare the data
with LZW compression, which makes it a bit faster, at least for the
download ;-) Recently the compressed geotiff files will take about 25-30
GB for the "HGT-SRTM-World" coverage.

The tiles have 6001 x 6001 pixels in EPSG:4326 projection. For the
reliefs I needed to reproject the data to other projections (EPSG:3857
or i.e. 3112) and I used gdalwarp for this with lanczos interpolation,
which gives me the best results for reliefs. Iin a graphical sense of
course, I don’t know if it would be also the case for other purpose with
lanczos.

-Yves

[HB-GRAL]

Post by HB-GRAL
When it’s useful and gdalchop can handle this I can prepare the data
with LZW compression, which makes it a bit faster, at least for the
download ;-) Recently the compressed geotiff files will take about 25-30
GB for the "HGT-SRTM-World" coverage.

Have to correct myself, it’s "only" around 16 or 17 GB of data,
compressed. The coverage of SRTM data is +- 60 degrees latitude, but I
will be able to add some other sources soon I hope, with same 5x5 merged
tiling.

btw. I didn’t change the naming convention of tiles like CGIAR data
does, so the tif names are i.e. "N40W010.tif" or "S10E135.tif" etc.
(lower left corner when I’m not wrong ;-)

-Yves

[Adrian Musceac]

Post by HB-GRAL
Hi Christian
Nice to hear that there are some plans (?) with extending gdalchop. As I
mentioned earlier I’m preparing 5x5 degree tiles of all the SRTM data in
geotiff format to use with terragear (just in case someone wants to give
it a try once, I use it for my reliefs and this 5x5 tiles are a side
effect of processing this data). A 5x5 degree geotiff uncompressed will
have around 72 MB.
When it’s useful and gdalchop can handle this I can prepare the data
with LZW compression, which makes it a bit faster, at least for the
download ;-) Recently the compressed geotiff files will take about 25-30
GB for the "HGT-SRTM-World" coverage.
The tiles have 6001 x 6001 pixels in EPSG:4326 projection. For the
reliefs I needed to reproject the data to other projections (EPSG:3857
or i.e. 3112) and I used gdalwarp for this with lanczos interpolation,
which gives me the best results for reliefs. Iin a graphical sense of
course, I don’t know if it would be also the case for other purpose with
lanczos.
-Yves

Hi Yves,

Having SRTM data available without the "normal but slightly restrictive" CGIAR
license would be a nice plus.
I do remember from FS2004 that there was a guy who had very detailed data
based on local topo maps and DEM's, his name was Jonathan de Ferranti and the
FS terrain was compiled by Holger Sandmann.
http://www.viewfinderpanoramas.org/dem3.html was the URL.
Can we get ahold of such detailed elevation maps, which far surpass SRTM,
especially for the Alps but other mountainous areas too? And could we use
this?

Cheers,
Adrian

[HB-GRAL]

Hi Adrian

I had some mail exchange with Jonathan Ferranti/Viewfinderpanorama two
years ago and he wrote me that he has no objection to my use of this
data for terragear purposes (I planned a small and personal scenery
project with this data two years ago and was sending a lot of requests
around the world, always asking if this could be used for a GPL
project). Generally Viewfinderpanorama needs credits for this of course,
but it looks like the data is available for FlightGear/TerraGear
projects, indeed.

To say it in words of Jonathan Ferranti:
"I hope that you find the data useful and that they will contribute to
the quality of your scenery."

Later I realised in discussion with Martin Spott that the World Scenery
Project is "well known" at Viewfinderpanorama and that I just replicated
what was already organized/prepared by Martin. It looks like the World
Scenery Project already made use of this data and that parts of the data
are also available via mapserver (not sure about, that has Martin Spott
to point out probably, we had some exchange about this data because of
my recent HGT-work some days/weeks ago, but I’m not sure about the
recent status of such directories).

Plans are (or should be) to have 1-2 servers where all this data is
available as a "data central" with some consistency like Martin Spott
always intended to have, and the flightgear mapserver with its
infrastructur will be primary, even when I plan to make it available
also here on my swiss servers (the fgx server, which has a good or at
least sufficent backbone too).

The origin reliefs will probably not be available on other servers. The
problem here is that it takes ~ 300 GB space at the end, and maybe it’s
also fairly experimental. But the tilecache produced with this data will
be available, so when someone wants to use that relief service for a map
it will show up in near future (hopefully!). Ehrm, this kind of work and
data has "Slow Food"-Labeling ;-)

-Yves

Post by HB-GRAL
Hi Christian
Nice to hear that there are some plans (?) with extending gdalchop. As I
mentioned earlier I’m preparing 5x5 degree tiles of all the SRTM data in
geotiff format to use with terragear (just in case someone wants to give
it a try once, I use it for my reliefs and this 5x5 tiles are a side
effect of processing this data). A 5x5 degree geotiff uncompressed will
have around 72 MB.
When it’s useful and gdalchop can handle this I can prepare the data
with LZW compression, which makes it a bit faster, at least for the
download ;-) Recently the compressed geotiff files will take about 25-30
GB for the "HGT-SRTM-World" coverage.
The tiles have 6001 x 6001 pixels in EPSG:4326 projection. For the
reliefs I needed to reproject the data to other projections (EPSG:3857
or i.e. 3112) and I used gdalwarp for this with lanczos interpolation,
which gives me the best results for reliefs. Iin a graphical sense of
course, I don’t know if it would be also the case for other purpose with
lanczos.
-Yves

Hi Yves,
Having SRTM data available without the "normal but slightly restrictive" CGIAR
license would be a nice plus.
I do remember from FS2004 that there was a guy who had very detailed data
based on local topo maps and DEM's, his name was Jonathan de Ferranti and the
FS terrain was compiled by Holger Sandmann.
http://www.viewfinderpanoramas.org/dem3.html was the URL.
Can we get ahold of such detailed elevation maps, which far surpass SRTM,
especially for the Alps but other mountainous areas too? And could we use
this?
Cheers,
Adrian
------------------------------------------------------------------------------
LogMeIn Rescue: Anywhere, Anytime Remote support for IT. Free Trial
Remotely access PCs and mobile devices and provide instant support
Improve your efficiency, and focus on delivering more value-add services
Discover what IT Professionals Know. Rescue delivers
http://p.sf.net/sfu/logmein_12329d2d
_______________________________________________
Flightgear-scenery mailing list
https://lists.sourceforge.net/lists/listinfo/flightgear-scenery