Virtual File System

Author:

Jeff McKenna

Contact:

jmckenna at gatewaygeomatics.com

Last Updated:

2023-07-30

Virtual File System

Starting with GDAL version 1.8.0, it is possible to read and display “virtual” VECTOR and RASTER files, such as to a remote file by pointing to a URL, or a compressed local .zip file that contains an OGR data source, and display that in MapServer. Useful mapfile parameters include /vsizip/, /vsigzip/, /vsitar/, /vsicurl/

More Information on Virtual File System

  • GDAL’s virtual file systems instructions

  • Blog post by initial developer Even Rouault

  • GDAL’s old Wiki page (might be outdated, but it is still a good source of information)

Display Compressed File Directly in MapServer

The following example uses a compressed shapefile, “test.zip”, that contains 3 files:

test.shp
test.dbf
test.zip

Step 1: Verify Format is “Virtual” Enabled

Execute ogrinfo –formats (for vector) or gdalinfo –formats (for raster) to check if the format lists a “v” beside the driver name, such as:

> ogrinfo --formats

Supported Formats:
  PCIDSK -raster,vector- (rw+v): PCIDSK Database File
  netCDF -raster,vector- (rw+s): Network Common Data Format
  PDF -raster,vector- (rw+vs): Geospatial PDF
  DB2ODBC -raster,vector- (rw+): IBM DB2 Spatial Database
  ESRI Shapefile -vector- (rw+v): ESRI Shapefile
  MapInfo File -vector- (rw+v): MapInfo File
  ...

The “(rw+v)” beside shapefile indicates that virtual files are supported.

Step 2: Test Access through ogrinfo/gdalinfo

Use ogrinfo with /vsizip/ to summarize the features of the zipped shapefile. Follow the OGR Vector Layers Through MapServer document examples for how to use the ogrinfo command with layers, such as:

> ogrinfo /vsizip/test.zip/test.shp test -summary

Note that “test.zip” exists in the same directory where we are running the ogrinfo command.

INFO: Open of `/vsizip/test.zip/test.shp'
      using driver `ESRI Shapefile' successful.

Layer name: test
Metadata:
  DBF_DATE_LAST_UPDATE=2000-03-17
Geometry: Polygon
Feature Count: 665
Extent: (-2750564.750000, -936638.500000) - (3583872.500000, 4673125.000000)
Layer SRS WKT:
(unknown)
AREA: Real (15.3)
PERIMETER: Real (15.3)
...

Step 3: Configure MapServer Layer

Once you have verified that you can read the features with ogrinfo (or gdalinfo for raster), then we can try creating a map in MapServer.

Warning

A full (absolute) path is required for the CONNECTION parameter, for these virtual layers.

LAYER
  NAME "test"
  TYPE POLYGON
  STATUS ON
  CONNECTIONTYPE OGR
  CONNECTION "/vsizip/C:/ms4w/apps/test/data/test.zip/test.shp"
  DATA "test"
  CLASS
    NAME "test"
    STYLE
      COLOR 240 240 240
      OUTLINECOLOR 199 199 199
    END
  END
END # layer

Note

You can omit the filename from the connection (test.shp) if there are no other files inside the zip than expected for a shapefile. When you omit it, the zip is recognized as a directory and it is the same logic as executing ogrinfo on a directory.

Step 4: Test your Mapfile

Verify that MapServer can generate a map image by using the map2img utility:

map2img -m mymapfile.map -o test.png -all_debug 5

msDrawMap(): rendering using outputformat named png (AGG/PNG).
msDrawMap(): WMS/WFS set-up and query, 0.000s
GDAL: GDALOpen(/vsizip/C:/ms4w/apps/test/data/test.zip/test.shp, this=07D691D8) succeeds as ESRI Shapefile.
Shape: 492 features read on layer 'test'.
GDAL: GDALClose(/vsizip/C:/ms4w/apps/test/data/test.zip/test.shp, this=07D691D8)
msDrawMap(): Layer 0 (test), 0.016s
msDrawMap(): Drawing Label Cache, 0.000s
msDrawMap() total time: 0.031s
msSaveImage(ttt.png) total time: 0.000s
msFreeMap(): freeing map at 06289688.

See also

Debugging MapServer

Display Remote Compressed File Directly in MapServer

We can also display an external compressed file, that is accessible through HTTP, using /vsicurl/ with /vsizip/

Test Access through ogrinfo/gdalinfo

> ogrinfo -ro /vsizip/vsicurl/https://download.osgeo.org/mapserver/docs/vsicurl-test.zip/test.shp test -summary

Configure MapServer Layer

LAYER
  NAME "test"
  TYPE POLYGON
  STATUS ON
  CONNECTIONTYPE OGR
  CONNECTION "/vsizip/vsicurl/https://download.osgeo.org/mapserver/docs/vsicurl-test.zip/test.shp"
  DATA "test"
  CLASS
    NAME "test"
    STYLE
      COLOR 240 240 240
      OUTLINECOLOR 199 199 199
    END
  END
END # layer

Warning

Virtual layers have a “cost” / are not as fast to display in MapServer as regular layers.

Display Remote (RASTER) File Directly in MapServer

Remote raster files can also be accessed through HTTP, using /vsicurl/

Test Access through gdalinfo

> gdalinfo /vsicurl/https://download.osgeo.org/gdal/data/gtiff/small_world.tif

Driver: GTiff/GeoTIFF
Files: /vsicurl/https://download.osgeo.org/gdal/data/gtiff/small_world.tif
Size is 400, 200
Coordinate System is:
GEOGCS["WGS 84",
    DATUM["WGS_1984",
        SPHEROID["WGS 84",6378137,298.257223563,
            AUTHORITY["EPSG","7030"]],
        AUTHORITY["EPSG","6326"]],
    PRIMEM["Greenwich",0],
    UNIT["degree",0.0174532925199433],
    AUTHORITY["EPSG","4326"]]
Origin = (-180.000000000000000,90.000000000000000)
Pixel Size = (0.900000000000000,-0.900000000000000)
...

Configure MapServer Layer

LAYER
  NAME "test"
  TYPE RASTER
  STATUS ON
  DATA "/vsicurl/https://download.osgeo.org/gdal/data/gtiff/small_world.tif"
  CLASS
    NAME "test"
  END
END # layer

Tip

You can enable debugging of this virtual connection by using the CPL_CURL_VERBOSE config variable in your mapfile, such as:

MAP
  ...
  CONFIG "CPL_DEBUG" "ON"
  CONFIG "CPL_CURL_VERBOSE" "ON"
  ...
  LAYER
    ...
  END
END