As part of a research project I have generously been sent a copy of state road networks, in a NETWORK.gdb folder, with about 180 files inside it.
I (mistakenly) thought this was something I could import into PostGIS using shp2pgsql (No) or ogr2ogr (using the PGeo Driver, but if it is I can't find the appropriate file to do it with... trying
$ ogrinfo NETWORK.gdb/gdb
I just get FAILURE: Unable to open datasource `NETWORK.gdb/gdb' with the following drivers. -> GRASS -> ESRI Shapefile -> MapInfo File -> UK .NTF -> SDTS -> TIGER -> S57 -> DGN -> VRT -> REC -> Memory -> BNA -> CSV -> GML -> GPX -> KML -> GeoJSON -> Interlis 1 -> Interlis 2 -> GMT -> SQLite -> DODS -> ODBC -> PGeo -> OGDI -> PostgreSQL -> MySQL -> XPlane -> AVCBin -> AVCE00 -> Geoconcept
Is this another type of geodatabase... have I missed something? I don't seem to be able to open it with uDig or QGIS. I don't have any ESRI products (and I'm a student working remotely!) Is my only option to go back on bended knee and ask if they can export the data in a different format (I don't want to stretch a favour here - they say they only provide the data in their own format)
cheers
Ben
--
Ben Madin REMOTE INFORMATION
t : +61 8 9192 5455 f : +61 8 9192 5535 m : 0448 887 220 Broome WA 6725
Bended knee. You've been supplied with a "file base geodatabase" and the only things that will open it are ESRI products or things licensing ESRI products.
On Thu, Apr 30, 2009 at 6:32 PM, Ben Madin <b...@remoteinformation.com.au> wrote: > G'day all,
> As part of a research project I have generously been sent a copy of state > road networks, in a NETWORK.gdb folder, with about 180 files inside it.
> I (mistakenly) thought this was something I could import into PostGIS using > shp2pgsql (No) or ogr2ogr (using the PGeo Driver, but if it is I can't find > the appropriate file to do it with... trying
> Is this another type of geodatabase... have I missed something? I don't seem > to be able to open it with uDig or QGIS. I don't have any ESRI products (and > I'm a student working remotely!) Is my only option to go back on bended knee > and ask if they can export the data in a different format (I don't want to > stretch a favour here - they say they only provide the data in their own > format)
> cheers
> Ben
> --
> Ben Madin > REMOTE INFORMATION
> t : +61 8 9192 5455 > f : +61 8 9192 5535 > m : 0448 887 220 > Broome WA 6725
> Bended knee. You've been supplied with a "file base geodatabase" and > the only things that will open it are ESRI products or things > licensing ESRI products.
Or products that have been provided with enough instance examples to be able to reverse engineer. Manifold GIS is one such product. I don't have any GDB examples to check its ability to suck the data out.
Something I totally agree with. There was a brief discussion on "The Shapfile 2.0 Manifesto" (http://moreati.org.uk/blog/2009/03/01/shapefile-20-manifesto/) about what should replace the shapefile. The ESRI file based GeoDatatabase was discussion with Scott Morehouse himself telling everyone about a non ArcObjects based API that would open up interoperability etc (the usual smoke and mirrors). No talk about the format specs being donated to the public domain; nothing about how the API would be licensed. You will never see an FDO Provider from ESRI I would guess. More of the same "do it our way or the highway"....
> On Thu, Apr 30, 2009 at 6:32 PM, Ben Madin <b...@remoteinformation.com.au> wrote: >> G'day all,
>> As part of a research project I have generously been sent a copy of state >> road networks, in a NETWORK.gdb folder, with about 180 files inside it.
>> I (mistakenly) thought this was something I could import into PostGIS using >> shp2pgsql (No) or ogr2ogr (using the PGeo Driver, but if it is I can't find >> the appropriate file to do it with... trying
>> Is this another type of geodatabase... have I missed something? I don't seem >> to be able to open it with uDig or QGIS. I don't have any ESRI products (and >> I'm a student working remotely!) Is my only option to go back on bended knee >> and ask if they can export the data in a different format (I don't want to >> stretch a favour here - they say they only provide the data in their own >> format)
>> cheers
>> Ben
>> --
>> Ben Madin >> REMOTE INFORMATION
>> t : +61 8 9192 5455 >> f : +61 8 9192 5535 >> m : 0448 887 220 >> Broome WA 6725
>> Bended knee. You've been supplied with a "file base geodatabase" and the >> only things that will open it are ESRI products or things licensing ESRI >> products.
> Or products that have been provided with enough instance examples to be able > to reverse engineer. Manifold GIS is one such product. I don't have any GDB > examples to check its ability to suck the data out.
Simon,
Are you suggesting that Manifold GIS has reverse engineered the file geodatabase format? Can you provide any pointers supporting that? If those guys can reverse engineer it, then so could we given enough desire.
> Something I totally agree with. There was a brief discussion on "The > Shapfile 2.0 Manifesto" > (http://moreati.org.uk/blog/2009/03/01/shapefile-20-manifesto/) about what > should replace the shapefile. The ESRI file based GeoDatatabase was > discussion with Scott Morehouse himself telling everyone about a non > ArcObjects based API that would open up interoperability etc (the usual > smoke and mirrors). No talk about the format specs being donated to the > public domain; nothing about how the API would be licensed. You will never > see an FDO Provider from ESRI I would guess. More of the same "do it our > way or the highway"....
I too am disappointed that ESRI declared they would provide open access to file geodatabases, and then let the actually follow through drag on for years after they came into initial use. However, I do believe they are contemplating a reasonably open API - possibly source included. They have also been considering support for an FDO and/or OGR provider. I'm a bit vague on the current plan but they are aware of these possibilities and are interested in doing them. So don't be so sure about what will never happen.
Best regards, -- ---------------------------------------+----------------------------------- --- I set the clouds in motion - turn up | Frank Warmerdam, warmer...@pobox.com light and sound - activate the windows | http://pobox.com/~warmerdam and watch the world go round - Rush | Geospatial Programmer for Rent
I had to check the current version. Manifold, like OGR, only supports ESRI Personal Geodatabase (*.mdb) format. Not ESRI File Geodatabase format. See: http://www.manifold.net/info/formats.shtml
----- Original Message ----- From: "Frank Warmerdam" <warmer...@pobox.com> To: "PostGIS Users Discussion" <postgis-us...@postgis.refractions.net> Sent: Thursday, 28 May, 2009 10:23:02 GMT -08:00 US/Canada Pacific Subject: Re: [postgis-users] ESRI GDB format into PostGIS
Simon Greener wrote: > Paul et al,
>> Bended knee. You've been supplied with a "file base geodatabase" and the >> only things that will open it are ESRI products or things licensing ESRI >> products.
> Or products that have been provided with enough instance examples to be able > to reverse engineer. Manifold GIS is one such product. I don't have any GDB > examples to check its ability to suck the data out.
Simon,
Are you suggesting that Manifold GIS has reverse engineered the file geodatabase format? Can you provide any pointers supporting that? If those guys can reverse engineer it, then so could we given enough desire.
> Something I totally agree with. There was a brief discussion on "The > Shapfile 2.0 Manifesto" > (http://moreati.org.uk/blog/2009/03/01/shapefile-20-manifesto/) about what > should replace the shapefile. The ESRI file based GeoDatatabase was > discussion with Scott Morehouse himself telling everyone about a non > ArcObjects based API that would open up interoperability etc (the usual > smoke and mirrors). No talk about the format specs being donated to the > public domain; nothing about how the API would be licensed. You will never > see an FDO Provider from ESRI I would guess. More of the same "do it our > way or the highway"....
I too am disappointed that ESRI declared they would provide open access to file geodatabases, and then let the actually follow through drag on for years after they came into initial use. However, I do believe they are contemplating a reasonably open API - possibly source included. They have also been considering support for an FDO and/or OGR provider. I'm a bit vague on the current plan but they are aware of these possibilities and are interested in doing them. So don't be so sure about what will never happen.
Best regards, -- ---------------------------------------+----------------------------------- --- I set the clouds in motion - turn up | Frank Warmerdam, warmer...@pobox.com light and sound - activate the windows | http://pobox.com/~warmerdam and watch the world go round - Rush | Geospatial Programmer for Rent
> Are you suggesting that Manifold GIS has reverse engineered the file > geodatabase format? Can you provide any pointers supporting that? > If those guys can reverse engineer it, then so could we given enough > desire.
I'm using Manifold on daily basis. Not to import ESRI files. Just looked through all possible imports and couldn't find ESRI GDB file
Yes, Manifold has reverse-engineered Personal and Enterprise GeoDatabases. DOn't know how well.
Type "GeoDatabase" into Search in the Manifold Help.
But, the way it is done is that one uses Tools>Database Console tto go to the database (eg Access .mdb) file that holds the geodata. When Manifold opens it it will recognise the ESRI crap and allow you to import/link to it.
regards Simon
> Frank Warmerdam wrote:
>> Are you suggesting that Manifold GIS has reverse engineered the file >> geodatabase format? Can you provide any pointers supporting that? >> If those guys can reverse engineer it, then so could we given enough >> desire. > I'm using Manifold on daily basis. Not to import ESRI files. > Just looked through all possible imports and couldn't find ESRI GDB file
> I had to check the current version. Manifold, like OGR, only supports ESRI Personal Geodatabase (*.mdb) format. Not ESRI File Geodatabase format. See: http://www.manifold.net/info/formats.shtml
Personal and Enterprise (SDE) GeoDatabases and NOT File GeoDatabases the situation with 8.0.x. (It may change in the up and coming 9.0 beta.)
> ----- Original Message ----- > From: "Frank Warmerdam" <warmer...@pobox.com> > To: "PostGIS Users Discussion" <postgis-us...@postgis.refractions.net> > Sent: Thursday, 28 May, 2009 10:23:02 GMT -08:00 US/Canada Pacific > Subject: Re: [postgis-users] ESRI GDB format into PostGIS
> Simon Greener wrote: >> Paul et al,
>>> Bended knee. You've been supplied with a "file base geodatabase" and the >>> only things that will open it are ESRI products or things licensing ESRI >>> products.
>> Or products that have been provided with enough instance examples to be able >> to reverse engineer. Manifold GIS is one such product. I don't have any GDB >> examples to check its ability to suck the data out.
> Simon,
> Are you suggesting that Manifold GIS has reverse engineered the file > geodatabase format? Can you provide any pointers supporting that? > If those guys can reverse engineer it, then so could we given enough > desire.
>> Something I totally agree with. There was a brief discussion on "The >> Shapfile 2.0 Manifesto" >> (http://moreati.org.uk/blog/2009/03/01/shapefile-20-manifesto/) about what >> should replace the shapefile. The ESRI file based GeoDatatabase was >> discussion with Scott Morehouse himself telling everyone about a non >> ArcObjects based API that would open up interoperability etc (the usual >> smoke and mirrors). No talk about the format specs being donated to the >> public domain; nothing about how the API would be licensed. You will never >> see an FDO Provider from ESRI I would guess. More of the same "do it our >> way or the highway"....
> I too am disappointed that ESRI declared they would provide open access > to file geodatabases, and then let the actually follow through drag on > for years after they came into initial use. However, I do believe they > are contemplating a reasonably open API - possibly source included. They > have also been considering support for an FDO and/or OGR provider. I'm > a bit vague on the current plan but they are aware of these possibilities > and are interested in doing them. So don't be so sure about what will > never happen.
> Are you suggesting that Manifold GIS has reverse engineered the file > geodatabase format? Can you provide any pointers supporting that? > If those guys can reverse engineer it, then so could we given enough > desire.
I probably misread the initial email confusing File for Personal (Access) and Enterprise (SDE). As I have said in other emails, Manifold 8.0.x supports the latter but not, as yet, the former. The up and coming 9.0 beta might change this.
> I too am disappointed that ESRI declared they would provide open access > to file geodatabases, and then let the actually follow through drag on > for years after they came into initial use. However, I do believe they > are contemplating a reasonably open API - possibly source included. They > have also been considering support for an FDO and/or OGR provider. I'm > a bit vague on the current plan but they are aware of these possibilities > and are interested in doing them. So don't be so sure about what will > never happen.
I've blogged on this in reference to The Shapefile 2.0 Manifesto. My views on ESRI and File GeoDatabases are probably best summarised on that page.
ESRI's fascination with creating proprietary file formats is, quite frankly, childish. But great "command and control" marketing.
ESRI Geodatabases This topic continues the general introduction to geospatial data storage covered by the Data Storage Strategies and the Spatial DBMS topics. This topic should be read together with those two topics, which introduce terminology and concepts used in this topic as well. This topic breaks out information specific to ESRI use of databases for new types of ESRI geospatial storage for the convenience of ESRI users.
ESRI, a legacy GIS vendor, has introduced a variety of products that can work with geometry and attribute data stored in DBMS servers. These are not spatial DBMS servers as such are generally understood, but rather are software products and middleware software that manage data stored in real DBMS servers using either blob or geometry types if the DBMS is a spatial DBMS. It is a classic case of a GIS vendor providing their own spatial capabilities for a DBMS by using their own non-native geometry types together with supporting capabilities supplied by the GIS vendor.
Manifold can also work with non-native data stored in databases using ESRI conventions. In such cases Manifold will work with ESRI's geometry types and supporting metadata tables using ESRI conventions for compatibility with ESRI products.
Nomenclature
Many users are baffled by ESRI nomenclature when it comes to parsing the bewildering variety of marketing phrases ESRI has used to describe ESRI "geodatabase" formats. If you feel baffled, you are not alone. In a nutshell, ESRI at one point introduced the idea of storing geometry in DBMS using a format that more or less boiled down to storing shapefiles within blobs. This was done in a complex way using middleware called ArcSDE that worked with serious databases like Oracle, and it was also done in a somewhat simplified way in Personal Geodatabase products that used Access .MDB files and were later apparently updated to work with MSDE (a free version of Microsoft SQL Server) or with SQL Server Express. In recent years, the ArcSDE product name seems to have been dropped by ESRI: more recent versions of this technology have been packaged as part of the ArcGIS product family and have been referred to as geodatabases.
All such storage methods are technically similar and are generally referred to as SDE geodatabase formats or as Personal geodatabase formats when in the somewhat simplified form that uses Access .MDB for file-based storage. Since all such formats are similar or derived from ArcSDE, they are referred to by Manifold documentation as ESRI SDE or as ESRI Geodatabase or as Personal Geodatabase data sources, the various terms being used interchangeably, regardless of which file format or DBMS system is used to store the data.
The terms are used interchangeably because some ESRI users come from a long ArcSDE tradition and don't realize that "geodatabase" is the new term for the same old thing, while some newer ESRI users might not realize that their "geodatabase" is really SDE technology with a new name. Because of the confusion caused by ESRI names for their SDE and their Personal technology being so similar, Manifold documentation will often refer to SDE and Personal geodatabases to underline that a particular capability is available whether one is working with either SDE geodatabases or the somewhat simpler Personal geodatabases.
Manifold can also connect to ESRI SDE and Personal geodatabase data sources for full read / write / edit capability. The only limitation is that unlike all other work with all other spatial DBMS, Manifold will not create new SDE or new Personal geodatabases, nor will Manifold add new drawings to an existing geodatabase. If we already have drawings in an SDE or Personal geodatabase, Manifold will happily import or link to those drawings. We can edit those drawings, adding new objects and deleting or editing old objects and in general perform whatever operation we like. For example, we could link to an existing drawing in a geodatabase and then copy and paste objects from some other drawing into that drawing. However, we cannot create new drawings or new geodatabases.
ESRI ArcSDE / ArcGIS / Personal Geodatabases
ESRI's ArcSDE product stores drawing geometry and other GIS data within ordinary, non-spatial DBMS servers. ESRI products refer to such data as geodatabases or SDE data sources (see notes on nomenclature above).
Technically, one can organize an SDE data source on almost any database. However, since this can not be done in a database-neutral fashion and since setting up an SDE data source involves creating database-specific objects, SDE data sources are only organized on big-name databases explicitly supported by ESRI, such as IBM DB2, IBM Informix, Microsoft SQL Server 2000 or 2005 and Oracle. SDE data sources using Access .mdb appear to have been replaced with SQL Server Express 2005. Personal geodatabases seem to be found almost exclusively within .mdb files.
Manifold knows to look for SDE or Personal geodatabase data if we use Database Console to connect to a database. Manifold usage of SDE or Personal geodatabase data sources uses Database Console as the primary interface and includes:
· Connecting to an SDE data source. · Listing the drawings in an SDE data source in Database Console. · Importing drawings. · Linking drawings in read-write mode.
When importing or linking drawings from SDE or Personal geodatabase data sources Manifold will fetch the coordinate systems (projections) in use from ESRI metadata. Importing or linking a drawing assigns it the coordinate system stored on the data source.
Manifold will convert ESRI style objects within the SDE database into Manifold equivalents. For example, reading data from an SDE geodatabase reads parametric curves, flattening them into lines with straight line segments. As of the current writing Manifold does not accept "multipoint" values, although this capability is expected to be added in future editions.
Although Manifold can connect to an existing SDE database, read (import) drawings, write drawings, link drawings and edit drawings, Manifold will not export new drawings to an SDE database nor will Manifold create a new SDE database.
<si...@spatialdbadvisor.com> wrote: > Frank and Piotr,
> Yes, Manifold has reverse-engineered Personal and Enterprise GeoDatabases. DOn't know how well.
> Type "GeoDatabase" into Search in the Manifold Help.
> But, the way it is done is that one uses Tools>Database Console tto go to the database (eg Access .mdb) file that holds the > geodata. When Manifold opens it it will recognise the ESRI crap and allow you to import/link to it.
> regards > Simon >> Frank Warmerdam wrote:
>>> Are you suggesting that Manifold GIS has reverse engineered the file >>> geodatabase format? Can you provide any pointers supporting that? >>> If those guys can reverse engineer it, then so could we given enough >>> desire. >> I'm using Manifold on daily basis. Not to import ESRI files. >> Just looked through all possible imports and couldn't find ESRI GDB file
postgis-users-requ...@postgis.refractions.net> wrote: > Date: Sat, 30 May 2009 09:46:46 +0700 > From: Bruce Foster <gis.fos...@gmail.com> > Subject: Re: [postgis-users] ESRI GDB format into PostGIS > To: PostGIS Users Discussion <postgis-us...@postgis.refractions.net> > Message-ID: > <c97454c50905291946m76c5e023n74842b3fabcc5...@mail.gmail.com> > Content-Type: text/plain; charset=ISO-8859-1
> HI
> this could help;
I think it could, except that it is not entirely accurate :)
> ESRI Geodatabases
...
> Nomenclature
> Many users are baffled by ESRI nomenclature when it comes to parsing > the bewildering variety of marketing phrases ESRI has used to describe > ESRI "geodatabase" formats. If you feel baffled, you are not alone.
Amen to that!
> In > a nutshell, ESRI at one point introduced the idea of storing geometry > in DBMS using a format that more or less boiled down to storing > shapefiles within blobs. This was done in a complex way using > middleware called ArcSDE that worked with serious databases like > Oracle, and it was also done in a somewhat simplified way in Personal > Geodatabase products that used Access .MDB files and were later > apparently updated to work with MSDE (a free version of Microsoft SQL > Server) or with SQL Server Express. In recent years, the ArcSDE > product name seems to have been dropped by ESRI: more recent versions > of this technology have been packaged as part of the ArcGIS product > family and have been referred to as geodatabases.
Yep
> All such storage methods are technically similar and are generally > referred to as SDE geodatabase formats or as Personal geodatabase > formats when in the somewhat simplified form that uses Access .MDB for > file-based storage. Since all such formats are similar or derived from > ArcSDE, they are referred to by Manifold documentation as ESRI SDE or > as ESRI Geodatabase or as Personal Geodatabase data sources, the > various terms being used interchangeably, regardless of which file > format or DBMS system is used to store the data.
Here, there is no mention of FileGDB, which is what the discussion was about.
> The terms are used interchangeably because some ESRI users come from a > long ArcSDE tradition and don't realize that "geodatabase" is the new > term for the same old thing, while some newer ESRI users might not > realize that their "geodatabase" is really SDE technology with a new > name. Because of the confusion caused by ESRI names for their SDE and > their Personal technology being so similar, Manifold documentation > will often refer to SDE and Personal geodatabases to underline that a > particular capability is available whether one is working with either > SDE geodatabases or the somewhat simpler Personal geodatabases.
This is incorrect. ArcSDE is, in a nutshell, the middleware that acts as a data access layer that is used by the GeoDatabase. The GeoDatabase is the term used to encapsulate the collections of technologies that include Topology, Geometric Networks, Network Datasets, Annotation, Representation Layers, Versioning, editing behavior, Disconnected Editing, Replication, Raster Catalogs, etc etc. So Personal GeoDatabase (mdb files), FileGDB (.gdb files), an Enterprise GeoDatabase (databases accessed through ArcSDE) are just implementations of the ESRI idea of what "GeoDatabase" entails.
> Manifold can also connect to ESRI SDE and Personal geodatabase data > sources for full read / write / edit capability.
This is the scary part as well as a bold statement. If you don't use any of the complex dataset types (i.e. you just use Simple Features within the ESRI environment), then "read/write/edit" is very simple to implement. Just write SQL directly (insome cases), stored procedures, use the ArcSDE client libraries, whatever.
However, "Full read / write / edit capability" would mean that you understand 100% all the different components inside the GeoDatabase and have implemented routines that cause edits to dirty, percolate, update, etc the different tables/rows involved in all GeoDatabase abstractions. In fact, within ESRI, nobody is able to claim that they understand the entire GeoDatabase (OK well, maybe only one person - and no, it is not Scott Morehouse). The reason is not technical difficulty, it is just a moving target: the behavior changes and since the format is not "published" it is easy to tweak and change within ArcGIS versions as long as the ArcObjects API is tweaked accordingly. No open spec, easy to change within versions, no need to explain it to anybody.
Hence, why there is no public FileGDB specification.
To do this, it would mean to open the box and let everyone know all the "precious ESRI secrets about the other complex types" and there hasn't been any "strong business argument" to cause enough presure inside ESRI to make that happen. Now I and (hopefully) you understand why open formats are a "good thing", but the philosophy behind it has not struck ESRI management strong enough. **My*** interpretation of the idea of "open architecture" inside ESRI (and I may be wrong since it has been a few years since I left my Redlands job) is that open interoperability is reached from the webservice tier and that there is no need for openess at any other level of the architecture stack.
> On Fri, May 29, 2009 at 6:38 AM, Simon Greener > <si...@spatialdbadvisor.com> wrote: > > Frank and Piotr,
> > Yes, Manifold has reverse-engineered Personal and Enterprise > GeoDatabases. DOn't know how well.
> > Type "GeoDatabase" into Search in the Manifold Help.
> > But, the way it is done is that one uses Tools>Database Console tto go to > the database (eg Access .mdb) file that holds the > > geodata. When Manifold opens it it will recognise the ESRI crap and allow > you to import/link to it.
> > regards > > Simon > >> Frank Warmerdam wrote:
> >>> Are you suggesting that Manifold GIS has reverse engineered the file > >>> geodatabase format? Can you provide any pointers supporting that? > >>> If those guys can reverse engineer it, then so could we given enough > >>> desire.
Frank W.: years ago when the first talks about the Open FileGDB API started, I personally suggested a GDAL/OGR driver as the primary venue for this open API. I know that the primary engineer dedicated to FileGDB contacted you directly with an inquiry about this topic. I know this, because he stopped by my office and we talked about it. If there is some momentum/need to reverse-engineer FileGDB (for Simple Features), I would recommend to start by contacting him directly. He is an extremely friendly fellow and will no doubt get you pointers that would help creating the open source driver a much easier task.
Lots of good, sensible comment. I particularly agree about SDE vs GeoDatabase. I believe Manifold GIS understands the small handful of tables in SDE that go way back to 1.0 but does not understand the GDB_* tables added at 8.x or the versioning tables. But I will check.
> Now I and (hopefully) you understand why open formats are a > "good thing", but the philosophy behind it has not struck ESRI management > strong enough.
I just am not one who ascribes to this viewpoint any more for spatial data management within enterprise, though I do agree we need specialised interchange formats where nothing exists in the IT world for data interchagne.
Is anyone seriously arguing that we need to force Oracle, Microsoft or IBM to make their storage formats public so programmers can create APIs and hack their way into the data that is stored within? How many GIS programmers here have gone to PostgreSQL's documentation to work out how a table is stored?
ESRI's problem is that they believe spatial data has an inherent logic to it that forces them to create new and wonderful (effectively) database formats to manage spatial (and attribue) data. They continue to force a divide into data management within organisations into two camps "spatial" (ie "mine") and "attribute" (ie "yours").
As Chris Date and Hugh Darwen wrote in their book "Foundation for Future Database Systems: The Third Manifesto".
“What we are saying is that, in the relational world, a domain is a data type, system- or user-defined, whose values are manipulable soley by means of the operators defined for the type in question (AND WHOSE INTERNAL REPRESENTATION CAN BE ARBITRARILY COMPLEX BUT IS HIDDEN FROM THE USER).” [My emphasis]
What Date and Darwen say for the "relational world" applies to all database technologies: what we need for access is logical interfaces not a dependence on accessing data by knowing the physical storage formats.
ESRI's efforts deliver neither open logical access technologies (eg spatial SQL via open JDBC implementations) or open physical formats.
Just my 4c worth.
S.
On Tue, 02 Jun 2009 06:13:32 +1000, Ragi Y. Burhum <rbur...@gmail.com> wrote:
> On Sat, May 30, 2009 at 12:00 PM, < > postgis-users-requ...@postgis.refractions.net> wrote:
>> Date: Sat, 30 May 2009 09:46:46 +0700 >> From: Bruce Foster <gis.fos...@gmail.com> >> Subject: Re: [postgis-users] ESRI GDB format into PostGIS >> To: PostGIS Users Discussion <postgis-us...@postgis.refractions.net> >> Message-ID: >> <c97454c50905291946m76c5e023n74842b3fabcc5...@mail.gmail.com> >> Content-Type: text/plain; charset=ISO-8859-1
>> HI
>> this could help;
> I think it could, except that it is not entirely accurate :)
>> ESRI Geodatabases
> ...
>> Nomenclature
>> Many users are baffled by ESRI nomenclature when it comes to parsing >> the bewildering variety of marketing phrases ESRI has used to describe >> ESRI "geodatabase" formats. If you feel baffled, you are not alone.
> Amen to that!
>> In >> a nutshell, ESRI at one point introduced the idea of storing geometry >> in DBMS using a format that more or less boiled down to storing >> shapefiles within blobs. This was done in a complex way using >> middleware called ArcSDE that worked with serious databases like >> Oracle, and it was also done in a somewhat simplified way in Personal >> Geodatabase products that used Access .MDB files and were later >> apparently updated to work with MSDE (a free version of Microsoft SQL >> Server) or with SQL Server Express. In recent years, the ArcSDE >> product name seems to have been dropped by ESRI: more recent versions >> of this technology have been packaged as part of the ArcGIS product >> family and have been referred to as geodatabases.
> Yep
>> All such storage methods are technically similar and are generally >> referred to as SDE geodatabase formats or as Personal geodatabase >> formats when in the somewhat simplified form that uses Access .MDB for >> file-based storage. Since all such formats are similar or derived from >> ArcSDE, they are referred to by Manifold documentation as ESRI SDE or >> as ESRI Geodatabase or as Personal Geodatabase data sources, the >> various terms being used interchangeably, regardless of which file >> format or DBMS system is used to store the data.
> Here, there is no mention of FileGDB, which is what the discussion was > about.
>> The terms are used interchangeably because some ESRI users come from a >> long ArcSDE tradition and don't realize that "geodatabase" is the new >> term for the same old thing, while some newer ESRI users might not >> realize that their "geodatabase" is really SDE technology with a new >> name. Because of the confusion caused by ESRI names for their SDE and >> their Personal technology being so similar, Manifold documentation >> will often refer to SDE and Personal geodatabases to underline that a >> particular capability is available whether one is working with either >> SDE geodatabases or the somewhat simpler Personal geodatabases.
> This is incorrect. ArcSDE is, in a nutshell, the middleware that acts as a > data access layer that is used by the GeoDatabase. The GeoDatabase is the > term used to encapsulate the collections of technologies that include > Topology, Geometric Networks, Network Datasets, Annotation, Representation > Layers, Versioning, editing behavior, Disconnected Editing, Replication, > Raster Catalogs, etc etc. So Personal GeoDatabase (mdb files), FileGDB (.gdb > files), an Enterprise GeoDatabase (databases accessed through ArcSDE) are > just implementations of the ESRI idea of what "GeoDatabase" entails.
>> Manifold can also connect to ESRI SDE and Personal geodatabase data >> sources for full read / write / edit capability.
> This is the scary part as well as a bold statement. If you don't use any of > the complex dataset types (i.e. you just use Simple Features within the ESRI > environment), then "read/write/edit" is very simple to implement. Just write > SQL directly (insome cases), stored procedures, use the ArcSDE client > libraries, whatever.
> However, "Full read / write / edit capability" would mean that you > understand 100% all the different components inside the GeoDatabase and have > implemented routines that cause edits to dirty, percolate, update, etc the > different tables/rows involved in all GeoDatabase abstractions. In fact, > within ESRI, nobody is able to claim that they understand the entire > GeoDatabase (OK well, maybe only one person - and no, it is not Scott > Morehouse). The reason is not technical difficulty, it is just a moving > target: the behavior changes and since the format is not "published" it is > easy to tweak and change within ArcGIS versions as long as the ArcObjects > API is tweaked accordingly. No open spec, easy to change within versions, no > need to explain it to anybody.
> Hence, why there is no public FileGDB specification.
> To do this, it would mean to open the box and let everyone know all the > "precious ESRI secrets about the other complex types" and there hasn't been > any "strong business argument" to cause enough presure inside ESRI to make > that happen. Now I and (hopefully) you understand why open formats are a > "good thing", but the philosophy behind it has not struck ESRI management > strong enough. **My*** interpretation of the idea of "open architecture" > inside ESRI (and I may be wrong since it has been a few years since I left > my Redlands job) is that open interoperability is reached from the > webservice tier and that there is no need for openess at any other level of > the architecture stack.
>> On Fri, May 29, 2009 at 6:38 AM, Simon Greener >> <si...@spatialdbadvisor.com> wrote: >> > Frank and Piotr,
>> > Yes, Manifold has reverse-engineered Personal and Enterprise >> GeoDatabases. DOn't know how well.
>> > Type "GeoDatabase" into Search in the Manifold Help.
>> > But, the way it is done is that one uses Tools>Database Console tto go to >> the database (eg Access .mdb) file that holds the >> > geodata. When Manifold opens it it will recognise the ESRI crap and allow >> you to import/link to it.
>> > regards >> > Simon >> >> Frank Warmerdam wrote:
>> >>> Are you suggesting that Manifold GIS has reverse engineered the file >> >>> geodatabase format? Can you provide any pointers supporting that? >> >>> If those guys can reverse engineer it, then so could we given enough >> >>> desire.
> Frank W.: years ago when the first talks about the Open FileGDB API > started, I personally suggested a GDAL/OGR driver as the primary venue for > this open API. I know that the primary engineer dedicated to FileGDB > contacted you directly with an inquiry about this topic. I know this, > because he stopped by my office and we talked about it. If there is some > momentum/need to reverse-engineer FileGDB (for Simple Features), I would > recommend to start by contacting him directly. He is an extremely friendly > fellow and will no doubt get you pointers that would help creating the open > source driver a much easier task.
"Geometry is just another data type" - one of my favourite mantras for years.
I'm not so sure that ESRI "forces" the divide between spatial and attribute but they are driving the spatial - it's the "be-all-and-end-all" of what they do according to Jack.
It is sooo tempting to fiddle the data model to make the application modelling easier. ESRI caught this disease with INFO and, it seems, just can't quite shake it off. But, as you say Simon, why should they? It works for them.
The first GIS I worked with was GeoVision (a geometry engine sitting on top of Oracle). When I first encountered ArcINFO I was horrified at the INFO "database" but bewitched and beguiled by the Arc geometry engine.
So, I'm interested Simon: "Do you think the ESRI Workspace XML has any future as a 'logical interface'?
[mailto:postgis-users-boun...@postgis.refractions.net] On Behalf Of Simon Greener Sent: Tuesday, 2 June 2009 10:40 AM To: PostGIS Users Discussion Subject: Re: [postgis-users] ESRI GDB format into PostGIS
Ragi,
Lots of good, sensible comment. I particularly agree about SDE vs GeoDatabase. I believe Manifold GIS understands the small handful of tables in SDE that go way back to 1.0 but does not understand the GDB_* tables added at 8.x or the versioning tables. But I will check.
> Now I and (hopefully) you understand why open formats are a > "good thing", but the philosophy behind it has not struck ESRI management > strong enough.
I just am not one who ascribes to this viewpoint any more for spatial data management within enterprise, though I do agree we need specialised interchange formats where nothing exists in the IT world for data interchagne.
Is anyone seriously arguing that we need to force Oracle, Microsoft or IBM to make their storage formats public so programmers can create APIs and hack their way into the data that is stored within? How many GIS programmers here have gone to PostgreSQL's documentation to work out how a table is stored?
ESRI's problem is that they believe spatial data has an inherent logic to it that forces them to create new and wonderful (effectively) database formats to manage spatial (and attribue) data. They continue to force a divide into data management within organisations into two camps "spatial" (ie "mine") and "attribute" (ie "yours").
As Chris Date and Hugh Darwen wrote in their book "Foundation for Future Database Systems: The Third Manifesto".
“What we are saying is that, in the relational world, a domain is a data type, system- or user-defined, whose values are manipulable soley by means of the operators defined for the type in question (AND WHOSE INTERNAL REPRESENTATION CAN BE ARBITRARILY COMPLEX BUT IS HIDDEN FROM THE USER).” [My emphasis]
What Date and Darwen say for the "relational world" applies to all database technologies: what we need for access is logical interfaces not a dependence on accessing data by knowing the physical storage formats.
ESRI's efforts deliver neither open logical access technologies (eg spatial SQL via open JDBC implementations) or open physical formats.
> "Geometry is just another data type" - one of my favourite mantras for > years.
Yep, mine too. But I've been a database guy since my Computer Science degree days: now well over 25 years ago!
> I'm not so sure that ESRI "forces" the divide between spatial and attribute > but they are driving the spatial - it's the "be-all-and-end-all" of what > they do according to Jack.
What I meant to say is that it seems to me that if you swallow the Enterprise GeoDatabase thing and attempt to implement it then you will come up against the IT world which doesn't think in such terms or use such narrow, stovepipe, technology. My last paid employer's developers used ERWin to develop around 5 database applications (property rights, asset management, conservation, inventory and supply chain) based on Oracle + spatial. The datamodels had all sorts of modelling that ESRI simply has never "allowed" over the years: multiple spatial columns per table; multiple spatial objects per column; circles etc.
What ESRI is saying to IT Departments is that that approach is wrong: if the datamodel contains spatial data it is a GeoDatabase and must be modelled in Visio and deployed into their database, middle-tier and client-tier technologies.
As if.
I have asked ESRIlites over the years how many GeoDatabases are fully designed, specified and implemented? Shall we start a survey on what people in this forum think?
> It is sooo tempting to fiddle the data model to make the application > modelling easier. ESRI caught this disease with INFO and, it seems, just > can't quite shake it off. But, as you say Simon, why should they? It works > for them.
Sure it works for them and the audience they have captured with their education, sales and marketing efforts. But I can't help think that to the bigger, broader, IT world they are a niche player. We worry about them because we ply our trade in a market segment dominated by the ESRI zeitgeist.
> The first GIS I worked with was GeoVision (a geometry engine sitting on top > of Oracle). When I first encountered ArcINFO I was horrified at the INFO > "database" but bewitched and beguiled by the Arc geometry engine.
Mate, me too! I hated INFO (preferred Genamap's theoretical and flawed approach) and have come over 25 years of use to respect the algorithmns in the Arc fortran code.
> So, I'm interested Simon: "Do you think the ESRI Workspace XML has any > future as a 'logical interface'?
I know nothing about the Workspace XML format. My experience has been more with the XML Schema of the GeoDatabase. Now with this format the problem is that everything inside the XML is couched in ESRI terminology and speak. Also, I am not convinced that the format is as open as they seem to imply. Try and get a hold of the XSD. There is a legal gray area around this format that scares me. Also there is no evidence to say that ESRI will play fair: the shapefile format is OPEN but they never published the spatial index file formats (Gee, I wonder why?). Finally, very few ESRIlites know how to generate such a document such that it contains:
1. Schematics; 2. Instance Data 3. Schematics and Instance data.
Frankly, ESRI never plays fair: why do we waste our time looking over our shoulders?
Well, maybe not that much, but a bit. The reason I'm watching is because I'm hanging out to be able to afford an FME licence.
> Shall we start a survey on what people in this forum think?
I think that would be a great idea.
I've fully designed and specified four 'enterprise' spatial databases and a dozen or so large, complex 'application' ones. None have been implemented as designed because the spatial experts (yes 'ESRIlites') all end up saying "ArcMap will cark it." (BTW: I still believe that there is nothing to stop you implementing a multi-geometry column table in a geodatabase; you just have to manage it yourself because the ESRI application infrastructure won't. Which, I think is your point. I say 'believe' because I haven't been able to get anyone imaginative enough to prove its scalability in a 'real' environment.)
People like the model simple because it makes the application of it simple. Lots more work, but simple. ("Waddaya mean 'I should write a query for my map layer'? Make me a new table!") Gee - I really am getting crankier as I get older!
So, back to the issue: I spent most of the 90's working on Data Interchange and ended up completely disillusioned. One interchange format will never cover all the bases. The dream now is for an Open Spatial ETL suite. (GDAL-OGR-FDO with muscle.)
None of this helps Ben with his problem but I do think it points to an answer for your question " why do we waste our time looking over our shoulders?" - Because they're the elephant in the room.
Cheers, and thanks for your insights on the workspace.xml
[mailto:postgis-users-boun...@postgis.refractions.net] On Behalf Of Simon Greener Sent: Tuesday, 2 June 2009 4:53 PM To: PostGIS Users Discussion Subject: Re: [postgis-users] ESRI GDB format into PostGIS
Alan ,
> My 3.25 cents (the $AUS is rising)
That much?!!
> "Geometry is just another data type" - one of my favourite mantras for > years.
Yep, mine too. But I've been a database guy since my Computer Science degree days: now well over 25 years ago!
> I'm not so sure that ESRI "forces" the divide between spatial and attribute > but they are driving the spatial - it's the "be-all-and-end-all" of what > they do according to Jack.
What I meant to say is that it seems to me that if you swallow the Enterprise GeoDatabase thing and attempt to implement it then you will come up against the IT world which doesn't think in such terms or use such narrow, stovepipe, technology. My last paid employer's developers used ERWin to develop around 5 database applications (property rights, asset management, conservation, inventory and supply chain) based on Oracle + spatial. The datamodels had all sorts of modelling that ESRI simply has never "allowed" over the years: multiple spatial columns per table; multiple spatial objects per column; circles etc.
What ESRI is saying to IT Departments is that that approach is wrong: if the datamodel contains spatial data it is a GeoDatabase and must be modelled in Visio and deployed into their database, middle-tier and client-tier technologies.
As if.
I have asked ESRIlites over the years how many GeoDatabases are fully designed, specified and implemented? Shall we start a survey on what people in this forum think?
> It is sooo tempting to fiddle the data model to make the application > modelling easier. ESRI caught this disease with INFO and, it seems, just > can't quite shake it off. But, as you say Simon, why should they? It works > for them.
Sure it works for them and the audience they have captured with their education, sales and marketing efforts. But I can't help think that to the bigger, broader, IT world they are a niche player. We worry about them because we ply our trade in a market segment dominated by the ESRI zeitgeist.
> The first GIS I worked with was GeoVision (a geometry engine sitting on top > of Oracle). When I first encountered ArcINFO I was horrified at the INFO > "database" but bewitched and beguiled by the Arc geometry engine.
Mate, me too! I hated INFO (preferred Genamap's theoretical and flawed approach) and have come over 25 years of use to respect the algorithmns in the Arc fortran code.
> So, I'm interested Simon: "Do you think the ESRI Workspace XML has any > future as a 'logical interface'?
I know nothing about the Workspace XML format. My experience has been more with the XML Schema of the GeoDatabase. Now with this format the problem is that everything inside the XML is couched in ESRI terminology and speak. Also, I am not convinced that the format is as open as they seem to imply. Try and get a hold of the XSD. There is a legal gray area around this format that scares me. Also there is no evidence to say that ESRI will play fair: the shapefile format is OPEN but they never published the spatial index file formats (Gee, I wonder why?). Finally, very few ESRIlites know how to generate such a document such that it contains:
1. Schematics; 2. Instance Data 3. Schematics and Instance data.
Frankly, ESRI never plays fair: why do we waste our time looking over our shoulders?
> Well, maybe not that much, but a bit. The reason I'm watching is because I'm > hanging out to be able to afford an FME licence.
Have you tried that doyen of the open spatial world: Talend's Spatial Data Integrator?
>> Shall we start a survey on what people in this forum think?
> I think that would be a great idea.
Well folks, there's the question:
How many (Enterprise/Personal) GeoDatabases are fully designed, specified and implemented such that they can use the Code Generation Wizard to generate code that implements “behaviours” within the ArcGIS technology?
> I've fully designed and specified four 'enterprise' spatial databases and a > dozen or so large, complex 'application' ones. None have been implemented as > designed because the spatial experts (yes 'ESRIlites') all end up saying > "ArcMap will cark it." (BTW: I still believe that there is nothing to stop > you implementing a multi-geometry column table in a geodatabase; you just > have to manage it yourself because the ESRI application infrastructure > won't. Which, I think is your point. I say 'believe' because I haven't been > able to get anyone imaginative enough to prove its scalability in a 'real' > environment.)
There isn't anything stopping you if you are prepared to do a bit of "behind the scenes" manipulation of ArcSDE. During the beta for SDE 2.0 (the first ESRI Inc produced) I showed how to have two spatial columns per table. Can be done.
> People like the model simple because it makes the application of it simple. > Lots more work, but simple. ("Waddaya mean 'I should write a query for my > map layer'? Make me a new table!") Gee - I really am getting crankier as I > get older!
You and I ought to take this off line as I have comments on this that might offend!
> So, back to the issue: I spent most of the 90's working on Data Interchange > and ended up completely disillusioned. One interchange format will never > cover all the bases. The dream now is for an Open Spatial ETL suite. > (GDAL-OGR-FDO with muscle.)
Again, see Talend.
I've been hoping for Spatial SQL JDBC/ODBC/ADO etc etc drivers for years and years and years. Guess they might just never arise.
> None of this helps Ben with his problem but I do think it points to an > answer for your question " why do we waste our time looking over our > shoulders?" - Because they're the elephant in the room.
Even if this elephant wanted to be gentle (as their maketing says) they simply can't: not enough brain cells to drive the physical body. And I won't be in a hurry to be in the same room as them.
> Cheers, and thanks for your insights on the workspace.xml
I have plenty of samples and some XSLT-style processing if you are interested. But contact me off-line.
|
