Scripts/Streams

From J Wiki
Jump to navigation Jump to search

Notes

The streams verb can be used to list the NTFS File Streams associated with a file.

I became curious about streams after http://www.jsoftware.com/pipermail/general/2008-May/031902.html . A brief rummage on the web showed that trying to figure out the details of how to list stream names was a common topic.<<FootNote(Well, they're called resource forks on Mac classic, so no wonder they interest J programmers. -- B Jonas)>>

I found there were two ways to list stream names. The simplest works only with Windows Vista, using a common FindFirst..., FindNext... functional interface. The only wrinkle I had with this code was my initial exposure to Unicode-only Windows functions, for both parameter input and returned results.

The more complex way to get the stream names is to use the BackupRead, BackupSeek API. This method should work for any version of Windows from NT through Vista. It works in a fundamentally different way than the prior method. It is designed for use by NTFS backup programs. BackupRead reads all of the data associated with a NTFS file as a sequential stream of bytes, including the stream header information. This code takes advantage of the fact that BackupSeek will not seek beyond the start of the next block of stream header information. The code does a very large seek to skip over any non-header data in the stream to get to the next stream header.

Usage

Example usage: 

   streams jpath'~temp/test.txt'
+---+---------------------------------------------+
|4  |::$DATA                                      |
+---+---------------------------------------------+
|168|:SummaryInformation:$DATA                   |
+---+---------------------------------------------+
|3  |:a:$DATA                                     |
+---+---------------------------------------------+
|3  |:b:$DATA                                     |
+---+---------------------------------------------+
|12 |:c:$DATA                                     |
+---+---------------------------------------------+
|0  |:{4c8cc155-6c1e-11d1-8e41-00c04fb9386d}:$DATA|
+---+---------------------------------------------+

[{{#file: "stream.ijs"}} Download script: stream.ijs ] 

NB. Stream Utilities

NB. streams jpath'~temp/test.txt'

FindFirstStreamW=: 'kernel32 FindFirstStreamW > i *w i * i'&cd
FindNextStreamW=: 'kernel32 FindNextStreamW > i i *'&cd
FindClose=: 'kernel32 FindClose > i i'&cd
GetLastError=: 'kernel32 GetLastError > i'&cd
FormatMessage=: 'kernel32 FormatMessageA > i  i i i i *c i i'&cd

CloseHandleR=: 'kernel32 CloseHandle > i i'&(15!:0)
CreateFileR=: 'kernel32 CreateFileA > i *c i i * i i i'&(15!:0)
BackupRead=: 'kernel32 BackupRead > i i * i *i i i *'&(15!:0)
BackupSeek=: 'kernel32 BackupSeek > i i i i *i *i *'&(15!:0)

hdr4=: _4}.(1) 3!:1 (2-2)
hdr8=: _8}.(3) 3!:1 (2-2)

NB. typedef enum _STREAM_INFO_LEVELS {
NB.
NB.     FindStreamInfoStandard,
NB.     FindStreamInfoMaxInfoLevel
NB.
NB. } STREAM_INFO_LEVELS;

NB. typedef struct _WIN32_FIND_STREAM_DATA {
NB. 8  LARGE_INTEGER StreamSize;
NB. 4  WCHAR cStreamName[MAX_PATH + 36];
NB. } WIN32_FIND_STREAM_DATA,

NB. #define MAX_PATH 260

NB. #define ERROR_HANDLE_EOF 38L

FORMAT_MESSAGE_FROM_SYSTEM=:4096

HFILE_ERROR=: -1
NULL=: 0{a.

CREATE_ALWAYS=: 2
CREATE_NEW=: 1
FALSE=: 0
FILE_BEGIN=: 0
FILE_END=: 2
GENERIC_READ=: _2147483648
GENERIC_WRITE=: 1073741824
OPEN_ALWAYS=: 4
OPEN_EXISTING=: 3
TRUNCATE_EXISTING=: 5
NULLPTR=: <0

NB.*streams v lists file streams associated with a file
streams=: 3 : 0
sd=. (8+2*260+36)#'_'

NB. call the pre-Vista verb if FindFirstStreamW is not available.

try. r=.FindFirstStreamW (2 u: y,{.a.);(2-2);sd;2-2 catch. streamsbu y return. end.

if. HFILE_ERROR=r do. (WFM GetLastError '') return. end.
d=.1 2$getstreamdata sd
h=.r
while. 1 do.
 r=.FindNextStreamW h;sd
 if. 0=r do. break. end.
 d=. d,getstreamdata sd
end.
FindClose h
d
)

NB.*getstreamdata v returns the size and name of the stream.
NB. note:  this will fail on 32 bit systems if the size of the stream is larger than _1+2^31
getstreamdata=: 3 : 0
(3!:2 hdr8,8{.y);((i.&NULL){.])1&u: 6&u: 8&}.y
)

NB. typedef struct _WIN32_STREAM_ID {
NB. 4 DWORD dwStreamId;
NB. 4 DWORD dwStreamAttributes;
NB. 8 LARGE_INTEGER Size;
NB. 4 DWORD dwStreamNameSize;
NB. WCHAR cStreamName[ANYSIZE_ARRAY];
NB. } WIN32_STREAM_ID,

NB. #define ANYSIZE_ARRAY 1

NB.*streamsbu v lists file streams associated with a file using the pre-Vista method.
streamsbu=: 3 : 0
  try. fh=. CreateFileR (y,{.a.);(GENERIC_WRITE+GENERIC_READ);0;NULLPTR;OPEN_EXISTING;0;0 catch.
    (13!:11 '');(13!:12 '')
    return.
  end.
  if. fh=_1 do.
    WFM GetLastError ''
    return.
  end.

sd=. (4+4+8+4+2*260+36)#'_'
sdz=. ,2-2
sdx=. ,2-2
ss1=. ,2-2
ss2=. ,2-2
d=. 0 2$''
while. 1 do.
 r=.BackupRead fh;sd;(20);sdz;(2-2);(2-2);sdx
 if. 0=r do. streambad fh return. end.
 if. sdz=0 do. break. end.
 'sid sat ssz snz'=.shdr sd
 if. 0<snz do.
  r=.BackupRead fh;sd;(snz);sdz;(2-2);(2-2);sdx
  if. 0=r do. streambad fh return. end.
  d=.d,(<ssz),<1&u: 6&u: snz{.sd
 else.
  d=.d,(<ssz),<'::$DATA'
 end.
 if. ssz>0 do.
  r=.BackupSeek fh;(2-2);(2^30);ss1;ss2;sdx
 end.
end.
r=.BackupRead fh;sd;(20);sdz;(3-2);(2-2);sdx
CloseHandleR fh
d
)

streambad=: 3 : 0
x=.WFM GetLastError ''
CloseHandleR fh
x
)

NB.*shdr v Convert the numeric portion of the stream header.
NB. note:  this will fail on 32 bit systems if the size of the stream is larger than _1+2^31
shdr=: 3 : 0
(;4 4 8 4{."0&.> 1)cb;.1 ]20{.y
)

cb=: 3 : 0
select. #y
 case. 4 do. 3!:2 hdr4,y
 case. 8 do. 3!:2 hdr8,y
end.
)

WFM=: 3 : 0
m=.255#' '
r=.FormatMessage FORMAT_MESSAGE_FROM_SYSTEM;0;y.;0;m;255;0
r{.m
)

Note 'Test' NB. select and Ctrl+E
  load 'files dir format'
  ferase jpath'~temp/test.txt'
  'test' fwrite jpath'~temp/test.txt'
  'abc'  fwrite jpath'~temp/test.txt:a'
  '123'  fwrite jpath'~temp/test.txt:b'
  fread jpath'~temp/test.txt'
  fread jpath'~temp/test.txt:a'
  fread jpath'~temp/test.txt:b'
  dir jpath'~temp'
NB. Use Explorer to add Summary Properties to test.txt
  streams jpath'~temp/test.txt'
  hexdump fread jpath'~temp/test.txt:SummaryInformation'
)

See Also

  • Wikipedia NTFS Reference,

http://en.wikipedia.org/wiki/NTFS

  • MSDN How To Use NTFS Alternate Data Streams,

http://support.microsoft.com/kb/105763

  • MSDN Iterating NTFS Streams,

http://msdn.microsoft.com/en-us/magazine/cc163677.aspx

  • MSDN A Programmer's Perspective on NTFS 2000 Part 1: Stream and Hard Link,

http://msdn.microsoft.com/en-us/library/ms810604.aspx#ntfs5_topic3

Footnotes

<<FootNote>>


Contributed by David Mitchell

Retrieved from "https://code.jsoftware.com/mediawiki/index.php?title=Scripts/Streams&oldid=41622"