Indigresso Wiki

Open Source Stuff for DASH7

User Tools

Site Tools


opentag:otlib:veelite_core_h

Veelite Core

Veelite is the DASH7 Filesystem implementation used by OpenTag. Veelite is designed to work especially with microcontrollers, which tend to have Flash memory. In order to use the Flash memory reliably, the Veelite Core implements a flash wear leveling method as well as a method for mirroring frequently-used files in SRAM. Veelite is somewhat agnostic to the wear leveling, but it is not agnostic to mirroring – mirroring is a fundamental feature of Veelite itself.

A Little Bit About Flash

Flash memory is read-only-memory (ROM) that can be erased and written by following special processes. These processes are not usually efficient. Moreover, Flash memories tend to have erase/write cycle limits. 10,000 cycles is a typical limit for MCU-based flash memories. The reason why Flash memory cannot be arbitrarily written is because, the way it works, Flash can only modify bits from 1→0. Erasing sets all the bits to 1, after which any value can be written. Moreover, erasing must occur in blocks, so an entire block of Flash must be erased just to modify one byte. Typical MCU's have block sizes between 128 bytes and 2048 bytes.

Of course, the Veelite Core implementation in OpenTag works around a lot of these problems.

Veelite Memory Structure

Veelite specifies a 24 bit virtual addressing space, although in current practice it is limited to 16 bits, therefore a total of 65536 bytes of file data can be accommodated. The Veelite Core is responsible for converting these virtual memory addresses into physical memory addresses, and then reading or writing to them. When you are configuring an OpenTag build, it is extremely important to set up your linker file and your configuration headers so that the compiler and linker both know where to look for the physical memories – in other words, just make sure the addresses are the same in both the linker script and the configuration files. If you set up these headers incorrectly, OpenTag will almost certainly crash or freeze when you try to run it.

VWORM

This would generally stand for “Virtual Write Once Read Multiple [Memory]”. In Veelite it is the addressing space that almost always maps to Flash.

VSRAM

This would generally stand for “Virtual Static Random Access Memory”. In Veelite it is the addressing space that maps to RAM. VSRAM is used for file mirroring.

EEPROM

Veelite itself is agnostic about EEPROM. A VWORM implementation (in the Veelite Core) could potentially use EEPROM instead of Flash. The Veelite Core might also stash variables in EEPROM that it wants to be non-volatile. A good example, here, is the address mapping table, which is discussed later in this article.

Veelite Core Implementations

The Veelite Core is responsible for reading/writing data from virtual addresses out-of/into physical memory locations. It is also responsible for performing memory initialization at startup (sometimes necessary) and formatting. The implementation is also, almost certainly, going to need to have a mapping table between virtual and physical addresses. For certain types of implementations this could be extremely simple, but if using Flash memory for VWORM, it is usually much more complicated.

Veelite Cores that use Flash memory for VWORM generally need to implement a table-based address resolution method and a wear leveling method. The standard Veelite Core implementation for OpenTag (Veelite Core X2) is designed with access speed and memory efficiency in mind.

Flash Block Rotation

All current Veelite Core implementations that use Flash include a similar method for flash block rotation. Blocks that cannot be written-to reliably are rotated into a fallow heap, and a fresh block is rotated out of the fallow heap. Then this fresh block gets written-to.

Veelite Core X2

The X2 system works by a process of logical transformations (i.e. a data cipher), such that the data in the flash may not actually represent the file data unless you know the logical mapping template. Thus, the mapping table needs to be saved in ROM when the power goes down, such that it can be brought back up when power resumes. Otherwise the data in Veelite will be lost.

Veelite Core X2 operates on three sets of blocks: primary, ancillary, and fallow. Primary blocks are blocks in their initial state after startup or after “recombination” (the process by which a primary and ancillary block are unified). Eventually, as writes occur, the primary block is joined by an ancillary block, and the data represented by a single virtual address is actually a logical combination of the data on two corresponding physical addresses of the primary+ancillary block system. The fallow blocks are fresh blocks that get rotated-in whenever a new ancillary block is needed. Old primary blocks get discarded into the fallow set following recombination.

Tricks like saving the mapping table when voltage gets low and then shutting down, are suitable for providing high-reliabilty for X2, and the savings in power of using the much faster X2 method more than make up for not being able to utilize the last drops of battery power. Alternatively, some platforms contain integrated EEPROM, and, on these sorts of devices, the mapping table can be saved directly to EEPROM. This strategy, when available, provides bulletproof reliability to the X2 method.

The Common Platform Interface has provisions for implementing table saves on low voltage detection. In all officially supported platforms, there is a method implemented to perform this activity. Moreover, it should be stated that the X2 method is novel to OpenTag/Veelite, and it is quite remarkable in terms of performance. It is not only more compact than other flash-wear-leveling systems for microcontrollers, but it performs reads and writes much faster as well.

Veelite Core X1

The X1 system works by using part of each Flash block as a header for identifying which addresses in the block are able to take writes and which probably aren't. Veelite Core X1 is basically orphaned, because the X2 method is much faster. Users needing more reliability in X2 have typically found ways to put the X2 mapping table into non-volatile memory rather than use the X1 method. Thus, the X1 method is deprecated.

Using EEPROM

For applications with modest filesystem requirements, it may be possible to use a platform that contains enough EEPROM to store the entire contents of the VWORM filesystem. In this case, the implementation of the Veelite Core becomes very simple: basically just an arithmetic transformation between virtual and physical addresses.

OTlib/veelite_core.h

Here is the code for veelite_core.h. You may also refer to the doxygen code documentation.

#   include "OT_types.h"
#   include "OT_config.h"



/** @note Virtual Address Space:
  * OpenTag's Virtual Address Space for nonvolatile memory is, at the moment,
  * limited to 64K.  The addresses available are 0x0000 to 0xFFFE (0xFFFF is
  * prohibited).
  *
  * There are two sectors that may work differently: VEEPROM and VWORM.
  *
  * VEEPROM: (deprecated)
  * An address space that can easily accommodate reads and writes.
  * It is optimized for storing a relatively small number of frequently accessed
  * & changed variables (which in NAND Flash requires wear-leveling). In the 
  * standard VEEPROM implementation each virtual address points to 16 bits of 
  * data.  The constant "VEEPROM_SIZE" is set to 2x maximum number of 16 bit
  * variables supported.  It is not really byte-addressable, though. 
  * Performance will improve if this number is reduced to reflect the actual 
  * amount of variables used (or close to it).
  *
  * VWORM: 
  * An address space that is best for reading blocks of data.
  * It is optimized for "Write Once Read Multiple" (WORM) behavior and is best
  * for storing blocks of data that aren't written to very often.  Unlike a true
  * WORM memory VWORM can be written to whenever necessary, but in the case that
  * your platform uses NAND Flash the write process is slow.
  */



/** @typedef vas_loc
  * An enum to declare the position in virtual addressing space (vas)
  * vas_error=0, in_vworm, in_veeprom
  */
typedef enum { 
    vas_error = 0, 
    in_vworm, 
    in_veeprom,
    in_vsram
} vas_loc;



/** @typedef vaddr
  * A currently 16 bit virtual address.  Later upgradable to 32 bit.
  * NULL_vaddr is not 0 but 0xFFFF, because in NAND Flash this value represents
  * uninitialized space.
  */
typedef ot_u16 vaddr;  
#   define NULL_vaddr               0xFFFF

#   define VWORM_BASE_VADDR         (0x0000)
#   define VWORM_PAGESIZE           FLASH_PAGE_SIZE
#   define VWORM_NUM_PAGES          OTF_VWORM_PAGES
#   define VWORM_FALLOW_PAGES       OTF_VWORM_FALLOW_PAGES
#   define VWORM_PRIMARY_PAGES      (OTF_VWORM_PAGES - OTF_VWORM_FALLOW_PAGES)
#   define VWORM_MAXBLOCKSIZE       VWORM_PAGESIZE                                ///@todo bring into implementation
#   define VWORM_BLOCKSIZE          VWORM_PAGESIZE                                ///@todo bring into implementation
#   define VWORM_BLOCKINFO_OFFSET   VWORM_BLOCKSIZE                                 ///@todo bring into implementation
#   define VWORM_SIZE               (VWORM_NUM_PAGES * VWORM_BLOCKSIZE)             ///@todo bring into implementation
#   define VWORM_ALLOC              (VWORM_NUM_PAGES * VWORM_PAGESIZE)

#   define VSRAM_BASE_VADDR         ISF_MIRROR_VADDR
#   define VSRAM_SIZE               ISF_MIRROR_HEAP_BYTES
#   define VSRAM_ALLOC              VSRAM_SIZE

#   define VWORM_BASE_PHYSICAL      OTF_VWORM_START_ADDR
#   define VWORM_PHYSICAL_ADDR(VAL) (VWORM_BASE_PHYSICAL + (VAL) - VWORM_BASE_VADDR)
//#   define VSRAM_BASE_PHYSICAL      OTF_VSRAM_START_ADDR
//#   define VSRAM_PHYSICAL_ADDR(VAL) (VSRAM_BASE_PHYSICAL + (VAL) - VSRAM_BASE_VADDR)



/** memory_faults
  * After a function returns false, the reason is in memory_faults.
  */

// STATUS FLAGS
#   define MEM_OK               0x00
#   define MEM_HW_FAULT         0x01
#   define MEM_ADDR_FAULT       0x02
#   define MEM_VWORM_FAULT      0x04
extern ot_u8 memory_faults;



/** @brief Checks which address space the supplied virtual address is in.
  * @param v_addr : (ot_uint) Virtual Address
  * @retval vas_loc : position of the address
  * @ingroup Veelite
  */
vas_loc vas_check(vaddr addr);



/** @brief Formats VWORM memory
  * @param none
  * @retval ot_u8 :       non-zero on memory fault
  * @ingroup Veelite
  *
  * This functions wipes all blocks and resets the virtual block structure.
  */
ot_u8 vworm_format( );



/** @brief Initializes the VWORM memory system
  * @param none
  * @retval ot_u8 :     Non-zero on memory fault
  * @ingroup Veelite
  *
  * VWORM keeps a table of active blocks in local SRAM, backed up in VEEPROM.
  * vworm_init() loads the backed up data into local SRAM.  If you have just
  * formatted, you do not need to init.
  */
ot_u8 vworm_init( );



/** @brief Saves the state of the vworm system
  * @param none
  * @retval ot_u8       Non-zero on memory fault
  * @ingroup Veelite
  *
  * This is sometimes implemented as an empty wrapper, depending on the way the
  * VWORM system saves its parameters.  Nonetheless, it should be run prior to
  * shutting down the system, power-cut off, etc.
  */
ot_u8 vworm_save( );



/** @brief Reads 16 bits of data at the virtual address
  * @param addr : (vaddr) Variable virtual address
  * @retval ot_u16 : returned read data
  * @ingroup Veelite
  *
  * if given an odd v_addr (A1), vworm_read() will pull the 16 bits that
  * contain this byte (i.e. A1A0 or A0A1 depending on endian)
  */
ot_u16 vworm_read(vaddr addr);



/** @brief Write 16 bits to a virtual address in VWORM
  * @param addr : (vaddr) Variable virtual address to write onto
  * @param data : (ot_u16) 16 bits of data to write.
  * @retval ot_u8 : Non-zero on memory fault
  * @ingroup Veelite
  *
  * @note If the supplied addr is odd, only the upper/lower 8 bits are written
  * for little/big endian.
  *
  * vworm is often held in NAND flash, so the implementation for vworm_write()
  * will typically include some sort of mechanism to account for wear leveling
  * write performance.  Typically, one or more fallow blocks of NAND are used,
  * and vworm_write() will shuffle the vaddr look up table as it automatically
  * shuffles the blocks. 
  */
ot_u8 vworm_write(vaddr addr, ot_u16 data);



/** @brief Mark a 16 bit value to a virtual address in VWORM
  * @param addr : (vaddr) virtual address to mark
  * @param value : (ot_u16) 16 bit value
  * @retval ot_u8 : Non-zero on memory fault
  * @ingroup Veelite
  * 
  * vworm_mark() is similar functionally to vworm_write, and in some 
  * implementations it may be identical.  In implementations with NAND flash,
  * vworm_mark() does not reshuffle blocks, so it is not guaranteed to work.
  * It is typically used to quickly zero a small amount of data, since NAND 
  * writes can go "downhill" (1->0) easily but not "uphill" (0->1) without
  * additional processing.
  */
ot_u8 vworm_mark(vaddr addr, ot_u16 value);
ot_u8 vworm_mark_physical(ot_u16* addr, ot_u16 value);



/** @brief Returns a physical byte pointer to the virtual address in VWORM
  * @param v_addr : (ot_uint) Virtual Address
  * @retval ot_u8* : the physical pointer to VWORM
  * @ingroup Veelite
  *
  * @note Data in VWORM, in most cases, is NOT contiguous.  This function is
  *       here "just because."  Besides, it is useful for debugging.
  */
ot_u8* vworm_get(vaddr addr);



/** @brief Debugging function that prints out the state of the block table
  * @param none
  * @retval none
  * @ingroup Veelite
  */
void vworm_print_table();


/** @brief Wipe (erase) a block of data in VWORM
  * @param v_addr : (ot_uint) Base Virtual Address of somewhere in the block
  * @param wipe_span : (ot_uint) number of bytes to wipe, starting from v_addr
  * @retval ot_u8 : Non-zero on memory fault
  * @ingroup Veelite
  *
  * @note This writes 0xFFFF all the words in the span.  It is not a block 
  *       erase function.  It is mostly for debugging.
  */
ot_u8 vworm_wipeblock(vaddr addr, ot_uint wipe_span);
//ot_u8 vworm_wipeblock_physical(ot_u8* addr, ot_uint wipe_span);







/** VSRAM is implemented in SRAM or something like it, that is byte addressable
  * and can be written to or read from without any overhead.
  */

/** @brief Reads 16 bits of data at the virtual address
  * @param addr : (vaddr) Variable virtual address
  * @retval (ot_u16*) data returned at supplied address
  * @ingroup Veelite
  *
  * if given an odd v_addr (A1), vsram_read() will pull the 16 bits that
  * contain this byte (i.e. A1A0 or A0A1 depending on endian)
  */
ot_u16 vsram_read(vaddr addr);


/** @brief Mark a 16 bit value to a virtual address in VSRAM
  * @param addr : (vaddr) virtual address to mark
  * @param value : (ot_u16) 16 bit value
  * @retval ot_u8 : Non-zero on memory fault
  * @ingroup Veelite
  * 
  * In VSRAM, "write" and "mark" are the same thing.  There are also macros,
  * vsram_write() and vsram_write_physical(), that resolve to these functions.
  */
ot_u8 vsram_mark(vaddr addr, ot_u16 value);
ot_u8 vsram_mark_physical(ot_u16* addr, ot_u16 value);
#define vsram_write(VAL1, VAL2)             vsram_mark(VAL1, VAL2)
#define vsram_write_physical(VAL1, VAL2)    vsram_mark_physical(VAL1, VAL2)


/** @brief Returns a physical byte pointer to the virtual address in VSRAM
  * @param v_addr : (ot_uint) Virtual Address
  * @retval ot_u8* : the physical pointer to VSRAM
  * @ingroup Veelite
  */
ot_u8* vsram_get(vaddr addr);
    
opentag/otlib/veelite_core_h.txt · Last modified: 2012/03/08 19:21 by jpnorair