[Flightgear-scenery] Madeira test suite with jenkins

[HB-GRAL]

Hi all

Since some weeks there is a terragear testsuite running on my jenkins.
It’s called "Madeira" and covers only a very small piece of scenery.

Every day my servers pull terragear trunk, build it and produce a chunk
of scenery with the build. It could also pull newest airport data and
shapefiles of course. With this data jenkins builds the scenery and
pushs that to a svn repo (so the scenery output is versioned).

It's a working model I want to discuss about how to create custom
scenery. It’s a model where new scenery is created daily, chunk by
chunk, and where different slaves are involved in future and so use
accumulated and networked power to generate scenery. It’s also possible
to work only on cetralized scenery sources and let the slaves pull and
produce the output.

At the moment "Madeira" could become some kind of a scenery test suite
for scenery generation tools (terragear tools). Also new/changed/updated
data sources can be tested. Developers working on terragear can verify
their changes and debugging with output created on a independent scenery
generation machine, following some well documented rules. Cause it’s
versioned I think there is all control you will need, distribution via
svn included.

http://code.google.com/p/madeira/

Of course, this is just a stub at the moment, it’s there to show my
proposal in what direction scenery creation could go ... some developers
helped me to test it and try to use it now (many thanks to them!).
Further thoughts, help and discussion is very welcome, anytime.

Cheers, Yves

[HB-GRAL]

Post by HB-GRAL
Hi all
Since some weeks there is a terragear testsuite running on my jenkins.
It’s called "Madeira" and covers only a very small piece of scenery.
Every day my servers pull terragear trunk, build it and produce a chunk
of scenery with the build. It could also pull newest airport data and
shapefiles of course. With this data jenkins builds the scenery and
pushs that to a svn repo (so the scenery output is versioned).

Now the server pulls OSM data daily from geofabrik.de and import that to
my postgres/postgis database. Another job produce the ESRI shapefile
output here: http://downloads.fgx.ch/geodata/data/osm/madeira/

This makes it possible to edit OSM data with JOSM (and collaborating to
the OSM project too) and having this data available the next day for the
madeira test suite.

-Yves

[HB-GRAL]

Hi Yves,
I hope you're doing well.
________________________________

Post by HB-GRAL
Now the server pulls OSM data daily from geofabrik.de and import that to
my postgres/postgis database. Another job produce the ESRI shapefile
output here: http://downloads.fgx.ch/geodata/data/osm/madeira/

This is exactly the process also done on Sphere, on which Christian and Pete are working to generate scenery. Wouldn't it be a good idea to all work on the same infrastructure: sphere is where the scenery has originally been built ? It is also the place where the new terrain will be pushed from via Terrasync, and where Pete and Chris are working hard to try to cope with terrain generation failures.

Hi Olivier

Thanks, am well.

Since Martin Spott wrote to the list he will not provide scenery anymore
I didn’t realise sphere remains the one and only scenery central. The
work done by Pete and Chris for scenery generarion tools will work for
every scenery build and any server, so it’s not necessary to centralize
scenery building on a single server. And terrasync can be pointed to
every subversion repository anyway.

I made a proposal to de-centralize scenery building with ORGANIZED
jenkins builds a contemporary way and I want to discuss this approach,
I’m not interested in discussing historical prerequisites on scenery cray 1.

So I don't really get the interest of having another server with the same data and the same goal (don't take it bad, I appreciate what you're doing), I just don't understand why we don't focus on our historical infrastructure, rather that having multiple resources, it makes things quite hard to understand.

My server is a very simple developing server not provoding the same data
(compared to what sphere provides), and it’s not a production server.
Hey, I'm not doubling data and the "sphere team" is free to use my data
of course (do you use my cleaned SRTM data now? It’s public for months).

Scenery development is not the only project I follow with my (privately
funded) infrastructure, so please do not blame me to provide resources
for the community, even I do not follow the historical world scenery
approach.

-Yves

[Olivier]

Hi Yves,



________________________________
De : HB-GRAL <***@sablonier.ch>
Envoyé le : Dimanche 2 septembre 2012 22h47

I didn't know you wanted to send our private mail exchange back onto the list, with a new content for your replies.

Post by HB-GRAL
Since Martin Spott wrote to the list he will not provide scenery anymore
I didn’t realise sphere remains the one and only scenery central.

Martin said was it at the end of February or early March that he was not able to cope with all the submissions he was receiving from the whole world, which is - when you imagine the workload it represents - pretty understandable, Martin being just like us a human, and nearly the only person dealing with those submissions.
He never stopped managing and enhancing the infrastructure behind, mapserver, scenemodels, etc. There were quite a lot of exchanges here on the devel or in private showing he is still there and working ;-)
Now with the webforms import, this workload has somewhat lowered, despite the increase of 3D model submissions, so that's a good news for all.
Sphere having all the tools, data, and free CPU time (and Terrasync infrastructure) still looks at the good place for scenery housing and terrain generation and broadcast.

Post by HB-GRAL
The work done by Pete and Chris for scenery generarion tools will work for
every scenery build and any server, so it’s not necessary to centralize
scenery building on a single server. And terrasync can be pointed to
every subversion repository anyway.

Why not, but that would be interesting as a spare resource, but while sphere is there, I don't take the real interest of having two resources to work on, maintain, update, apart from building a spare machine or testing, which would be great. However I still think the most important for now is to have the toolchain working and able to generate worldwide data. This is the top priority and while this is not done, I, personally, won't take the time to focus on something else.
It's already hard enough to make people work on a common infrastructure to build the worldwide scenery.

Post by HB-GRAL

  (don't take it bad, I appreciate what you're doing),

so please do not blame me to provide resources

^^. See the above line, I don't think there is any blame in any of my sentences, I'm just trying to understand. Your private mail was more friendly.

Oliver

[HB-GRAL]

Hi Yves,
________________________________
Envoyé le : Dimanche 2 septembre 2012 22h47
I didn't know you wanted to send our private mail exchange back onto the list, with a new content for your replies.

Post by HB-GRAL
Since Martin Spott wrote to the list he will not provide scenery anymore
I didn’t realise sphere remains the one and only scenery central.

Martin said was it at the end of February or early March that he was not able to cope with all the submissions he was receiving from the whole world, which is - when you imagine the workload it represents - pretty understandable, Martin being just like us a human, and nearly the only person dealing with those submissions.
He never stopped managing and enhancing the infrastructure behind, mapserver, scenemodels, etc. There were quite a lot of exchanges here on the devel or in private showing he is still there and working ;-)
Now with the webforms import, this workload has somewhat lowered, despite the increase of 3D model submissions, so that's a good news for all.
Sphere having all the tools, data, and free CPU time (and Terrasync infrastructure) still looks at the good place for scenery housing and terrain generation and broadcast.

Post by HB-GRAL
The work done by Pete and Chris for scenery generarion tools will work for
every scenery build and any server, so it’s not necessary to centralize
scenery building on a single server. And terrasync can be pointed to
every subversion repository anyway.

Why not, but that would be interesting as a spare resource, but while sphere is there, I don't take the real interest of having two resources to work on, maintain, update, apart from building a spare machine or testing, which would be great. However I still think the most important for now is to have the toolchain working and able to generate worldwide data. This is the top priority and while this is not done, I, personally, won't take the time to focus on something else.
It's already hard enough to make people work on a common infrastructure to build the worldwide scenery.

Post by HB-GRAL

(don't take it bad, I appreciate what you're doing),

so please do not blame me to provide resources

^^. See the above line, I don't think there is any blame in any of my sentences, I'm just trying to understand. Your private mail was more friendly.
Oliver

Hi Olivier

Sorry for my harsh words, please do not take that personal. I see a lot
of progress at two places at the moment: the objects database frontends
and the terragear tools core.

For the objects database (and probably also for other scenery data
edits) I proposed to move to a gedjango application with free access for
all contributors without restrictions, i.e. like the OpenStreetMap
project works (works!), with "personal scenery developer responsability"
and the philosophy of more self control for users instead of a
complicated control mechanism which keeps sphere office full of pending
control work. This is some kind of my personal misinterpretation in case
it really helped to easy the workflow. I’m happy when this is the case
now, and I think this is mainly your work.

What I see at the moment is work done that was pending for years and
it’s a good progress of course for database submissions, but why to
enrich the historical interface only and not start with a complete new
and contemporary project in background? ;-)

The former sphere developers always stated it’s necessary to work with
this and that (i.e. php), but I don’t think it’s a good idea to build a
geo island with this, when ~80 percent of the geo world is working with
other techniques. What’s about geoserver, geodjango, qgis, mapnik etc. ?
Has this been evaluated by sphere? I offered work and help for that
direction (and as you noticed probably sphere uses already one small
little mapnik script coming from me). I really want to improve that
part, but I need the feeling being in a team. Are you helping me with that?

Former world scenery developers have the theory that I’m mainly
replicating what is already done. But that’s not what I do when you dive
deeper into what I’m proposing.

I still miss documentation for scenery database/tools on sphere (beside
of comments in scripts and automatically generated doc tables). That’s
why I imported osm data to postgres 9.1/postgis 1.5 myself and
documented how this is done. To be honest I didn’t realize that on
sphere it’s done exactly the same way with osm data, I didn’t ask
either, because I thought working on scenery data is cancelled there.
And also I don’t know who has access to sphere and who is working there
(beside of Martin Spott).

Now I started to work on a "scenery test suite" some weeks ago for two
reasons (third is fun). The first idea is that scenery tools developers
(and also texture and shader developers i.e.) can test their work
without comparing scenery output from their private machines only. At
least there is a small piece of scenery public, versioned and
comparable. The second idea is to have a demonstration/example of a
possible workflow for future scenery edits and world scenery generation,
distributed over a network and not sphere only (even here sphere could
act as "master").

The test suite scenery is built with jenkins and updated daily, I don’t
see jenkins installed on sphere nor do I think Martin Spott is going to
give me sphere as a scenery test tools slave ? ;-)

But it can be transferred to any server of course.

-Yves

[HB-GRAL]

Post by Olivier
Sphere having all the tools, data, and free CPU time (and Terrasync infrastructure) still looks at the good place for scenery housing and terrain generation and broadcast.

Not all the tools, i.e. Jenkins is missing for my approach. I had a
jenkins on my dev server installed, so that’s the reason I started
there. Now I’m in discussion with Martin Spott off-list and maybe we can
move this whole idea to sphere for testing, at least I hope it. It’s not
me saying I don’t want to use sphere resources ;-)

-Yves