Efficiently indexing with offset bitmaps for huge sets of slightly disordered sensor data

Takahiro Yamazaki, Takeru Inoue, Hiroshi Sato, Noriyuki Takahashi, Junichiro Takagi, Masateru Minami

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)

Abstract

With advances in sensor devices and networking technologies, it is expected that future networks will contain immense numbers of sensors that are capturing time-varying data. It is necessary to process queries over the data for analysis and to store the data for later use. For querying the data, data disorder is a common problem. Existing approaches use buffers to recover the order but there are problems in deciding buffer sizes and buffer residence times. To avoid these problems and to satisfy the need to store the data for later use, our approach writes indexed data rapidly to a stable disk. This approach, however, has a problem in that existing indexing methods are not up to the task; "sparse indexing" offers fast write-speeds but cannot handle disordered data, "dense indexing" can handle disordered data but its write-speed is inadequate. Our solution is a novel indexing method that can rapidly store slightly disordered time-series sensor data. Our proposal succeeds because it uses a novel data structure, "offset bitmaps", to extend the sparse indexing method. The offset bitmaps provide pointers that can handle delayed data and so allow them to be managed efficiently. Experiments show that our method has several advantages over conventional alternatives. We verify that the proposed method has 10 times the write speed of the existing dense indexing method.

Original languageEnglish
Title of host publication8th Asia-Pacific Symposium on Information and Telecommunication Technologies, APSITT 2010
Publication statusPublished - 2010
Externally publishedYes
Event8th Asia-Pacific Symposium on Information and Telecommunication Technologies, APSITT 2010 - Kuching
Duration: 2010 Jun 152010 Jun 18

Other

Other8th Asia-Pacific Symposium on Information and Telecommunication Technologies, APSITT 2010
CityKuching
Period10/6/1510/6/18

Fingerprint

Sensors
Data structures
Time series
Experiments

Keywords

  • Component
  • Data structure
  • Indexing
  • Sensor network

ASJC Scopus subject areas

  • Computer Networks and Communications

Cite this

Yamazaki, T., Inoue, T., Sato, H., Takahashi, N., Takagi, J., & Minami, M. (2010). Efficiently indexing with offset bitmaps for huge sets of slightly disordered sensor data. In 8th Asia-Pacific Symposium on Information and Telecommunication Technologies, APSITT 2010 [5532030]

Efficiently indexing with offset bitmaps for huge sets of slightly disordered sensor data. / Yamazaki, Takahiro; Inoue, Takeru; Sato, Hiroshi; Takahashi, Noriyuki; Takagi, Junichiro; Minami, Masateru.

8th Asia-Pacific Symposium on Information and Telecommunication Technologies, APSITT 2010. 2010. 5532030.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Yamazaki, T, Inoue, T, Sato, H, Takahashi, N, Takagi, J & Minami, M 2010, Efficiently indexing with offset bitmaps for huge sets of slightly disordered sensor data. in 8th Asia-Pacific Symposium on Information and Telecommunication Technologies, APSITT 2010., 5532030, 8th Asia-Pacific Symposium on Information and Telecommunication Technologies, APSITT 2010, Kuching, 10/6/15.
Yamazaki T, Inoue T, Sato H, Takahashi N, Takagi J, Minami M. Efficiently indexing with offset bitmaps for huge sets of slightly disordered sensor data. In 8th Asia-Pacific Symposium on Information and Telecommunication Technologies, APSITT 2010. 2010. 5532030
Yamazaki, Takahiro ; Inoue, Takeru ; Sato, Hiroshi ; Takahashi, Noriyuki ; Takagi, Junichiro ; Minami, Masateru. / Efficiently indexing with offset bitmaps for huge sets of slightly disordered sensor data. 8th Asia-Pacific Symposium on Information and Telecommunication Technologies, APSITT 2010. 2010.
@inproceedings{bd702c03211441b9b0166fa5c73be125,
title = "Efficiently indexing with offset bitmaps for huge sets of slightly disordered sensor data",
abstract = "With advances in sensor devices and networking technologies, it is expected that future networks will contain immense numbers of sensors that are capturing time-varying data. It is necessary to process queries over the data for analysis and to store the data for later use. For querying the data, data disorder is a common problem. Existing approaches use buffers to recover the order but there are problems in deciding buffer sizes and buffer residence times. To avoid these problems and to satisfy the need to store the data for later use, our approach writes indexed data rapidly to a stable disk. This approach, however, has a problem in that existing indexing methods are not up to the task; {"}sparse indexing{"} offers fast write-speeds but cannot handle disordered data, {"}dense indexing{"} can handle disordered data but its write-speed is inadequate. Our solution is a novel indexing method that can rapidly store slightly disordered time-series sensor data. Our proposal succeeds because it uses a novel data structure, {"}offset bitmaps{"}, to extend the sparse indexing method. The offset bitmaps provide pointers that can handle delayed data and so allow them to be managed efficiently. Experiments show that our method has several advantages over conventional alternatives. We verify that the proposed method has 10 times the write speed of the existing dense indexing method.",
keywords = "Component, Data structure, Indexing, Sensor network",
author = "Takahiro Yamazaki and Takeru Inoue and Hiroshi Sato and Noriyuki Takahashi and Junichiro Takagi and Masateru Minami",
year = "2010",
language = "English",
isbn = "9781424464135",
booktitle = "8th Asia-Pacific Symposium on Information and Telecommunication Technologies, APSITT 2010",

}

TY - GEN

T1 - Efficiently indexing with offset bitmaps for huge sets of slightly disordered sensor data

AU - Yamazaki, Takahiro

AU - Inoue, Takeru

AU - Sato, Hiroshi

AU - Takahashi, Noriyuki

AU - Takagi, Junichiro

AU - Minami, Masateru

PY - 2010

Y1 - 2010

N2 - With advances in sensor devices and networking technologies, it is expected that future networks will contain immense numbers of sensors that are capturing time-varying data. It is necessary to process queries over the data for analysis and to store the data for later use. For querying the data, data disorder is a common problem. Existing approaches use buffers to recover the order but there are problems in deciding buffer sizes and buffer residence times. To avoid these problems and to satisfy the need to store the data for later use, our approach writes indexed data rapidly to a stable disk. This approach, however, has a problem in that existing indexing methods are not up to the task; "sparse indexing" offers fast write-speeds but cannot handle disordered data, "dense indexing" can handle disordered data but its write-speed is inadequate. Our solution is a novel indexing method that can rapidly store slightly disordered time-series sensor data. Our proposal succeeds because it uses a novel data structure, "offset bitmaps", to extend the sparse indexing method. The offset bitmaps provide pointers that can handle delayed data and so allow them to be managed efficiently. Experiments show that our method has several advantages over conventional alternatives. We verify that the proposed method has 10 times the write speed of the existing dense indexing method.

AB - With advances in sensor devices and networking technologies, it is expected that future networks will contain immense numbers of sensors that are capturing time-varying data. It is necessary to process queries over the data for analysis and to store the data for later use. For querying the data, data disorder is a common problem. Existing approaches use buffers to recover the order but there are problems in deciding buffer sizes and buffer residence times. To avoid these problems and to satisfy the need to store the data for later use, our approach writes indexed data rapidly to a stable disk. This approach, however, has a problem in that existing indexing methods are not up to the task; "sparse indexing" offers fast write-speeds but cannot handle disordered data, "dense indexing" can handle disordered data but its write-speed is inadequate. Our solution is a novel indexing method that can rapidly store slightly disordered time-series sensor data. Our proposal succeeds because it uses a novel data structure, "offset bitmaps", to extend the sparse indexing method. The offset bitmaps provide pointers that can handle delayed data and so allow them to be managed efficiently. Experiments show that our method has several advantages over conventional alternatives. We verify that the proposed method has 10 times the write speed of the existing dense indexing method.

KW - Component

KW - Data structure

KW - Indexing

KW - Sensor network

UR - http://www.scopus.com/inward/record.url?scp=77956473621&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=77956473621&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:77956473621

SN - 9781424464135

BT - 8th Asia-Pacific Symposium on Information and Telecommunication Technologies, APSITT 2010

ER -