Section 4 - Performance Considerations

Production Throughput

Q) How fast does OrthoVista process Images? What is expected throughput in a production environment?
A) The processing time and image throughput depend very strongly on the computer hardware, project data configuration and processing options. Therefore, it is not meaningful to attempt to provide a specific number for these values. However, as a rule of thumb, end-to-end processing should take less than an hour per image. Sub-optimal processing may be indicated by large amounts of disk I/O (e.g. program swapping) and/or large amounts of idol CPU time.
The following examples should provide some insight into performance expectations.
NOTE that actual performance on specific hardware, with specific data sets, project configuration and processing parameters can often change these estimates by extreme amounts. For example, a change to one of the "Adaptive Feathering" parameter values can easily half or double processing time - therefore, the following examples CAN NOT BE COMPARED ONE TO THE OTHER. They are presented here as INDEPENDENT case studies. Actual performance in each situation can be STRONGLY affected by changing processing parameters and by other computer/network activity.
Example 1: Typical project consisting of approximately 50 color images scanned at 1000 dpi (approximately 230 MB per image). Image overlap approximately 15% in each direction. Typical processing options of "Hot Spot" correction, "Global Tilting" and "Adaptive Feathering". Computer hardware including an "Alpha processor" running Linux with a 500 MHz processor and local RAID disk storage. Typical total processing time is on the order of 30 minutes per images (e.g. all 50 images would be dodged, color corrected and automatically mosaicked in approximately 24 hours).
Example 2: A typical project consisting of approximately 40 images with 10-15% forward and side overlap. Each ortho image produced by rectification of approximately 6.5 inch by 4.2 inch "sweet spot". Computer hardware including an Intel 486 computer running Windows NT with 66 MHz processor and 256 MB RAM. Image data access via 100-Base-T network to SGI Origin 200 server. For B/W images, typical processing included occasional "HotSpot" corrections and systematic use of "Global Tilting" and "Adaptive Feathering". For B/W images (approximately 130 MB each), processing times for the complete set of 40 images were approximately 8-10 hours (e.g. approximately 15 minutes per image). For Color images (approximately 500 MB each), processing time was in the range of 14-15 hours.
Example 3: A project with 100 input images with 10-15% overlap in each direction. Computer hardware including an SGI Indigo 2 with R4400/200MHz processor, 256 MB RAM and 27 GB local disk space. Standard processing options including "HotSpot", "Global Tilting" and "Adaptive Feathering". 100 image job completed in approximately 65 hours (e.g. .approximately 40 minutes per image).
Q) I'm not getting the kind of performance indicated above. What might be happening?
A) Most often, slow performance is the result of an inappropriate value for the memory cache setting. Refer to Setting Cache Size for more information on this. A related problem is that computer memory (or processor time) are being consumed by other programs or tasks running on the platform (e.g. although there is enough physical memory, there may not be enough available memory). On some older SGI/IRIX systems, the OS had a memory leak which would consume memory - thereby continuously reducing available memory over the course of several days.
The second most common cause of very slow performance is the use of slow (or very busy) network connections to access data. Refer to Hardware I/O Speed for a description of network processing effects.
A third potential cause of slow processing can be the result of having changed processing options to a configuration which requires significantly more processing activity. The easiest way to check this is to replace the OrthoVista configuration file, "bigmo.cfg" with a fresh copy (e.g. from the distribution). Refer to section on OrthoVista Configuration File for more information.
Also, on some systems (notably this is required by Windows/NT), the distribution utilizes dynamic link libraries. In some cases, one or more of the libraries used by OrthoVista may have been overwritten by installation of other software. If you suspect this, you can reinstall OrthoVista to recover the required (DLL) libraries. Of course, this will likely cause problems with the other applications - however, not much you can do about that since it's the way Windows is designed.

Platform and Hardware

Q) What is performance expectation on different platforms?
A) See Section 2.

Cache Size

Q) How do I select the memory cache size?
A) Read on...
Q) What factors affect cache size requirements.
A) Cache requirements are increased by many factors in addition to image size. These include, the number of processing operations being performed, the image overlap configuration, and file data structures (e.g. tiled vs. striped storage schemes). These individual influences can interact in a complex manner.
In addition to caching data for I/O, OrthoVista uses cache space to store intermediate processing results. This involves computing values (radiometric corrections for example) in units which are natural for the data. A typical processing data unit may be an entire scanline or an entire input/output data tile.
The data caching structure can be though of as a set of 'layers'. For example, the first layer could be the input data cache (to avoid repeat access of disk to get adjacent pixels), the second layer could be a unit of image data rotated by a software function call, the third could be a cache for radiometric corrections which have been applied to the rotated patch, and the last layer could be the output data cache.
The ideal situation (common in practice) is when the geometry of each cache layer lines up so that the input cache unit, the data processing cache units and the output cache unit contain the same geometric area. For example, this would be the case for tiled tiff input, radiometric corrections only (no rotation) and tiled tiff output image. Note that by its very definition, a general rotation operation destroys this alignment..
The rotation of images can greatly increase cache requirements. For example if an input image is stored with a scanline data structure and it is rotated 1/4 turn (+/-90 degrees) and the output format is also a scanline structure, then the entire image would need to be cached to provide reasonable performance. A general rule of thumb is to use tiled data structures for images which must be rotated.
Lastly, when multiple images are combined into a single image (e.g. during mosaicking) the cache structure must store data for all the input images as well as the output image. Therefore, the amount of overlap present in the input imagery affects the cache size requirements.
Q) How much effect does image tile size (not mosaic sheet tile size) have on cache requirements?
A) This can be dramatic. Image data are accessed as stored units of image tile size or image scan line size.
Because the image tile/stripe is the 'quatum' unit of image data I/O, it means that this much data must be loaded into memory even if only a single pixel is required for a compuation. Therefore the cache data use efficiency would be highest if stripes/tiles contained only a single pixel! However, if that were the case, then the disk I/O overhead would become overwhelming. Thus, optimal tile/strip size is a balance between "big enough for good disk I/O performance" and "small enough to avoid loading unnecessary data into the cache.
For example: assume a color TIFF image, 12000 pixels wide with file format using stripe storage with 5 scan lines per strip. This means that image data are accessed (read and write) in units of 180000 bytes (5(lines)*12000(pixel)*3(rgb)) bytes at a time. (Note that color data are stored in memory as 4 bytes to greatly enhance speed - at the expense of increasing memory requirements by 4/3). Cache memory is therefore consumed in increments of 240000 bytes.
As another example: assume a greyscale image in TIFF tiled format with each tile size 128x128 pixel. In this case the data 'quantum' loaded into (or unloaded from) the memory cache is 16384 bytes. That means even if only one pixel is need, that 16384 bytes of data are read from file and placed in memory cache. Note that increasing the tile size to 512x512 pixels increases the size of the data cache 'quantum' 16-fold! to 262144 bytes.
The number of tiles/strips which must be _simultaneously_ present in cache memory depends strongly on the image configuration geometry. High overlap and image rotation _greatly_ increase the number of tiles/strips which must simultaneously be available in cache. Therefore, there can be a dramatic effect on performance related to the size of each tile.
As illustrated in the second example above, simply changing a TIFF image tile size from 128 to 512 increase overall memory cache requirements by 16 times!!!
Case Study: Reported Example:
Initially, this project was only able to mosaic 20 images of about 70 image block. If processing more than these 20 images at once the processing consideribley slows down. The problem is that OV needs too much cache to process the whole block at once.
I made a test and found the following.
1. Tile Size of the tiff images was 512
2. After having reduced the tile size from 512 to 128 I had no problem to run the whole block with all 70 tiles at once.
NOTE: If you have no or little control over the tile/strip size for OrthoVista input images, you can use OrthoVista itself to modify the tile/strip sizes. Just "process" the image data with no radiometric options, save the "adjusted" images (and of course specify the tile/strip size under image options) - all from the "Processing" dialog.

Q) What is the effect of having a large amount of RAM? I understand that it is mainly used for Image caching but I am unsure how much is optimal. For example, say I have a project with about 5 GB of input imagery. Will I reap benefits from having say 512 MB or KGB over 256 MB for such a job or does it really only help with very large amounts of input data? I guess that as much RAM as you can lay your hands on will benefit and allow you to increase the image cache size and thus gain a performance benefit. Am I correct in this judgment?
A) There are two critical memory considerations. (1) Cache size - used for 'paging through' the masses of data in a project; and (2) Heap memory - used for storing "stuff that always needs to be around". An example of type (1) data include pixel values. An example of type (2) memory includes image boundaries use to determine which images potentially overlap or cover a specific world location.
The cache needs to be "big enough". There is a HUGE penalty if it is too small - but there is virtually NO benefit if it is bigger than this minimum (until it becomes "too big" in which case there is a very real penalty resulting from the corresponding reduction of available heap memory). When you run out of cache memory (e.g. cache is too small), the program performance will slow by many orders of magnitude - the CPU will stay busy computing (and recomputing and recomputing and recomputing... data values that should have been available in cache after being computed the first time)
The available heap memory (essentially AVAILABLE physical memory less cache size) is used to store data which must remain 'instantly available' at any point in program execution. Requirements for heap memory scale roughly in proportion to the number of input images and there is a substantial amount of heap memory required per image. The actual amount is highly data dependent but can be a couple MB per image. If you run out of heap memory, the result is usually an unglamorous crash (often after slowing to a crawl, eating all the system swap space, and making the system unusable for a couple minutes while the executing program expires).
For some specifics, many people run OrthoVista on 512 MB systems using cache sizes in the range of 50-100 MB. However, if you are working with very large (e.g. wide) images, images with large internal tiles/scanlines, images with lots of overlap - then cache requirements may be larger. Also, if you are working with larger numbers of images in a project (lets say more than 50 or so), you may want more memory. You can relatively quickly determine a sample minimum for cache size (ignoring production pressures, etc...). Run a representative output mosaic tile from somewhere in the middle of the input image area. Note execution time (approximately - e.g. 1/4 hr, 3/4 hr, etc.), Then run with smaller cache sizes - when cache is too small, you'll probably get tired of waiting before you can measure the time to completion. This will give you some information for the data configurations you're using - but remember the dangers of "sample sizes of one" and allow for considerable latitude and safety margins for production. Also remember that these sizes are VERY dependent on project data (image formats, project geometry configurations, processing options, etc.).
For general hardware selection, physical memory on the order of 512 MB (per OrthoVista running) is probably a 'solid base configuration' for most purposes. A KGB (per OrthoVista running) would be a "very nice" system.
To be more specific, you can use diagnostic tools to look at OrthoVista's memory use during some of your production runs. Then you can gauge how much memory is being used for your typical production projects. If you try two projects of greatly different sizes (e.g. 4-8 images in one, 40-80 in the other), then you can get a quick estimate of the per image memory (heap) overhead for your data configuration. To first order - note the difference in memory use and divide by the difference in number of images.

Hardware CPU Speed

Q) I expect that with the appropriate cache size for a given amount of memory the most important factor on performance is the CPU speed. Is this correct? Is it better to invest in more system RAM rather than the fastest CPU for OrthoVista?
A) Faster CPU is always better. However, a certain minimum amount of memory is required to hold program data and the program cache data (see setting cache size). Once this minimum amount of memory is available, then additional memory has little or no effect. However, CPU speed always had an effect. Therefore, a general advice is first buy "enough" memory, then put money into processor speed.

Hardware I/O Speed

Q) I have found OrthoVista not to be terribly dependent on fast I/O, unless you get the cache size wrong and end up swapping. That said, I expect that it would still benefit from the overall system performance boost you get from having fast hard disk subsystem. I have found that local storage of images rather than going over the network also helps, even though our server and network is very fast. What are your thoughts here? Am I correct in my assumptions?
A) Yes! OrthoVista is primarily CPU bound most of the time (we haven't measured, but probably 85-95% of it's time is computation for typical projects - provided the data being processed are available when needed. If there is data I/O latency (e.g. network latency), then the I/O can become a much more significant part of the processing time. In general, we recommend local data storage - also, at least on Linux systems, you can benefit from the OS using free memory as a large I/O cache.
Q) How does use of a networked data server affect OrthoVista performance?
A) In general you will obtain the best performance if input and output data are stored on a fast local disk system. Any competition for disk access (e.g. by other processes) can slow processing since OrthoVista must frequently go into a "wait state" until requested pixel data become available. This is particularly true if data are being accessed over a network. In this case, network speed and latency are added to disk access time. In addition, other network traffic can interfere significantly with data transmission to/from OrthoVista processing. Many sites are using well designed and tuned networks for OrthoVista processing. However, in other cases, a severely overloaded network can bring OrthoVista processing nearly to a halt.
Q) How do I know if network is slowing OrthoVista processing?
A) Create a small test block (e.g. 4 images). First process it over the network with input and output data stored on a remote system. Then make a copy of the image (and support) data on the local system and process it again with the exact same processing options. In both cases, run this as the only task on the processing computer. The difference in processing time will be an estimate of the extra time required for network data I/O.

Program "Hangs" or Aborts

Q) After completing processing of single image adjustments (e.g. hotspot corrections), program seems to hang. What is happening?
A) In many cases, the program is not actually hung. Rather it is making progress but only extremely slowly. This can happen because data are being swapped out of the cache and the program must recompute data that should have been stored in the cache. This occurs if the cache size is set too small and the program must recompute pixel values many times over (often computing the same value 10's of thousands of times for each pixel). The solution is to increase the cache size - but BEWARE that too large a cache can cause other, even worse memory problems. See Setting Cache Size.
Q) Program is running very slowly - may appear 'hung'. Disk lights (or system monitoring software) shows lots of disk activity. What is happening?
A) It may be that the cache size is too large. If the cache is too large to fit in available memory (allowing for the requirements of non-cached data, OS, OS data, and other programs which may be running), then it may get swapped. Since the program is designed assuming that cache data are 'instantly' available, the speed penalty for swapping the cache is absolutely enormous. See Setting Cache Size.
Q) Program dies ("core dump" on unix, "memory access error" or often "Dr. Watson's" error on NT). What is happening?
A) OrthoVista, like all programs, requires certain amounts of 'dynamically' allocated memory while it is running. This is used to hold the program executable itself, and also to store transitory data. If there is not enough memory available on the system, the program can not continue to run. This problem is most often associated with the cache size being set too big. See Setting Cache Size.
Q) Performing "hot spot" balancing with "color adjustment" DISabled causes a crash. Is there a workaround?
A) This is an identified problem with Versions 2.2.1 and 2.2.2. There is no effective workaround other than to process with this option ENabled or to utilize a newer (e.g. 2.2.3 or later) or an older version.