Overview¶

Path	PyPoE/poe/patchserver.py
Version	1.0.0a0
Revision	$Id: a774b7a7a40fa3180a0f93d38afd9f292931e676 $
Author	Omega_K2

Description¶

Utility functions and classes for connecting to the PoE patch server and downloading files from it.

Agreement¶

See PyPoE/LICENSE

Documentation¶

Public API¶

class PyPoE.poe.patchserver.Patch(master_server='172.65.204.172', master_port=12995)[source]¶

Bases: object

Class that handles connecting to the patching server and downloading files from the patching server.

Variables:	patch_url (str) – Base patch url for the current PoE version. This does not point to a specific, load-balanced server patch_cdn_url (str) – Load-balanced patching url including port for the current PoE version. sock_fd (fd) – Socket file descriptor for connection to patch server

__init__(master_server='172.65.204.172', master_port=12995)[source]¶

Automatically fetches patching urls on class creation.

Note

Parameter shouldn’t be required to be changed; if the servers change please create a pull request/issue on Github.

Parameters:	master_server (str) – Domain or IP address of the master patching server master_port (int) – Port to use when connecting to the master patching server

download(file_path, dst_dir=None, dst_file=None)[source]¶

Downloads the file at the specified path from the patching server.

Any intermediate directories for the write paths will be automatically created.

Parameters:

file_path (str) – path of the file relative to the content.ggpk root directory
dst_dir (str) –
Write the file to the specified directory.

The target directory is seen as the root directory, thus the file will be written according to it’s file_path

Mutually exclusive with the dst_file argument.
dst_file (str) –
Write the file to the specified location.

Unlike dst_dir this will ignore any naming conventions from file_path, so for example Data/Mods.dat could be written to C:/HelloWorld.txt

Mutually exclusive with the 'dst_dir argument.

Raises:

ValueError – if neither dst_dir or dst_file is set
ValueError – if the HTTP status code is not 200 (and it wasn’t raised by urllib)

download_raw(file_path)[source]¶

Downloads the raw bytes.

Parameters:	file_path (str) – path of the file relative to the content.ggpk root directory
Returns:	the raw contents of the file in bytes
Return type:	bytes
Raises:	`ValueError` – if the HTTP status code is not 200 (and it wasn’t raised by urllib)

update_patch_urls()[source]¶

Updates the patch urls from the master server.

Open a connection to the patchserver, get webroot details, detach from socket, and store socket file descriptor as sock_fd

Note

Recreate socket object later with: socket_fd_open()

When finished, destroy socket with: socket_fd_close(). Equivalent is called in __del__()

version¶

Retrieves the game version from the url.

Returns:	The gama version in x.x.x.x format. The first 3 digits match the public known versions, the last is internal scheme for the a/b/c patches and hotfixes.
Return type:	str

class PyPoE.poe.patchserver.PatchFileList(patch, socket_timeout=1.0)[source]¶

Bases: object

Class that retrieves file details from the patch server.

Example:

import PyPoE.poe.patchserver
patch = PyPoE.poe.patchserver.Patch()
patch_file_list = PyPoE.poe.patchserver.PatchFileList(patch)
patch_file_list.update_filelist(['Data'])

Note

Patch server protocol

Open TCP 12995 us.login.pathofexile.com
Client hello

push 01 04:

04 = patch proto 4

05 = patch proto 5
receive web root & backup web root
Client request folder details

push 03 00 folder_name_length folder_name_in_utf16_LittleEndian:

Root: 0300 00

Art: 0300 03 410072007400
receive single-depth item list for queried folder

proto 4 root example:

2 byte header:

0400

byte folder_name_length:

Root: 00

folder_name_length bytes folder_name:

Root: null

int list_length (number of items in folder):

00 00 00 17

list_of_items:

For each item:

2 byte item type:

0000 file

0100 folder (in content.ggpk)

byte item_name_length

item_name_length bytes UTF-16 item name

int(BE) file size in bytes

32 byte sha256sum

proto 5:

not understood: different datatypes, Endianness, and some values are compressed or encoded.

2 byte header:

0400

n-byte unknown:

?folder name length

?folder name

?list_length (number of items in folder):

list_of_items:

For each item:

n-byte unknown:

?item type

some missing int(BE) file size in bytes

some missing ?int(LE) item_name_length

some missing item_name_length-some_value bytes partial UTF-16 item name

32 byte sha256sum
client checks provided hashes / sizes against files, and Content.ggpk records for directories.
- directory hashes are SHA-256 of the concatenated SHA-256 hashes of the children.
- if file checksum fails, queue failed file for download.
- if directory checksum fails, get folder details for failed folder name (step 4) from patch server

Variables:

patch (Patch) – Store patch server details.
sock (socket) – Store socket, to use single connection for multiple queries.
sock_timeout (float) – Socket timeout value in seconds, for socket.socket.settimeout().
data (io.BytesIO) – Store server_data from socket, for processing in multiple methods.
directory (DirectoryNodeExtended) – Store patch file list data as PyPoE.poe.file.ggpk.DirectoryNode

__init__(patch, socket_timeout=1.0)[source]¶

Automatically fetch root file list on class creation.

Parameters:	patch (`Patch`) – A Patch object socket_timeout (float) – Socket timeout value in seconds, for `socket.socket.settimeout()`.

extract_varchar()[source]¶

Helper function to extract variable length string from data. String length is first byte of data.

Returns:	extracted variable length string
Return type:	str

read(read_length)[source]¶

Read length of data from data. Get and save more data from sock if length not met.

Parameters:	read_length (int) – Length of data to read
Returns:	Requested data
Return type:	bytes
Raises:	`EOFError` – If the TCP stream returned by the patch server ends unexpectedly

update_filelist(folders)[source]¶

Get file details for a folder from the patch server.

Stores data in directory

Patchserver works top down: PatchFileList().directory.children entries are not known until that directory is traversed.

Once a directory level is traversed, patchserver can be queried for next directory.

It will return item details for all items in queried directory.

Parameters:	folders (list) – The list of folders to get details for ?Only one level at a time
Raises:	`ValueError` – If folders list contains repeated folders `ValueError` – If root is requested alongside additional folders `KeyError` – If the patch server sends data not understood

class PyPoE.poe.patchserver.DirectoryNodeExtended(*args, **kwargs)[source]¶

Bases: PyPoE.poe.file.ggpk.DirectoryNode

Adds methods:

get_dict()

load_dict()

gen_walk()

directories¶

Returns a list of nodes which belong to directories

Returns:	list of `DirectoryNode` instances which contain a `DirectoryRecord`
Return type:	list[DirectoryNode]

extract_to(target_directory)¶

Extracts the node and its contents (including sub-directories) to the specified target directory.

Parameters:	target_directory (str) – Path to directory where to extract to.

files¶

Returns a list of nodes which belong to files

Returns:	list of `DirectoryNode` instances which contain a `FileRecord`
Return type:	list[DirectoryNode]

gen_walk(max_depth=-1, _depth=0)[source]¶

A depth first recursive generator for a DirectoryNode

Example:

for node, depth in patch_file_list.directory.gen_walk():
  try:
    name = node.record.name
  except:
    name = 'ROOT'
  print('{blank:>{width}}{name}'.format(
    name=name, width=depth, blank=''))

Parameters:	max_depth (int) – how many levels of children to walk
Returns:	(`DirectoryNodeExtended`, depth)
Return type:	tuple

get_dict(recurse=True)[source]¶

Get a dict of PyPoE.poe.file.ggpk.DirectoryNode record item details

Example:

from json import dump
dump_dict = patch_file_list.directory.get_dict()
dump_dict['version'] = patch_file_list.patch.version
file_handle = open('poe_file_details.json', 'w', encoding='utf-8')
dump(dump_dict, file_handle)
file_handle.close()

Parameters:	recurse (bool) – True = include children
Returns:	keys: hash name size type: folder or file Folders have children[]
Return type:	collections.OrderedDict

get_parent(n=-1, stop_at=None, make_list=False)¶

Gets the n-th parent or returns root parent if at top level. Negative values for n will iterate until the root is found.

If the make_list keyword is set to True, a list of Nodes in the following form will be returned:

[n-th parent, (n-1)-th parent, …, self]

Parameters:	n (int) – Up to which depth to go to. stop_at (DirectoryNode or None) – `DirectoryNode` instance to stop the iteration at make_list (bool) – Return a list of `DirectoryNode` instances instead of parent
Returns:	Returns parent or root `DirectoryNode` instance
Return type:	DirectoryNode

get_path()¶

Returns the full path

Returns:	Full path
Return type:	str

load_dict(node_dict, parent=None)[source]¶

Fill a DirectoryNode from a dict

Example:

import json
from collections import OrderedDict
file_handle = open('poe_file_details.json', 'r', encoding='utf-8')
file_dict = json.load(file_handle, object_pairs_hook=OrderedDict)
file_handle.close()

Parameters:	node_dict (collections.OrderedDict) – Ordered dict from `DirectoryNodeExtended.get_dict()`

name¶

Returns the name associated with the stored record.

Returns:	name of the file/directory
Return type:	str

search(regex, search_files=True, search_directories=True)¶

Parameters:	regex (re.compile()) – compiled regular expression to use search_files (bool) – Whether `FileRecord` instances should be searched search_directories (bool) – Whether `DirectoryRecord` instances should be searched
Returns:	List of matching :class:`DirectoryNode`s
Return type:	list[DirectoryNode]

walk(function)¶

Todo

function = None -> generator like os.walk (dir, [dirs], [files])

Walks over the nodes and it’s sub nodes and executes the specified function.

The function will be called with the following dictionary arguments:

node - DirectoryNode
depth - Depth

Parameters:	function (callable) – function to call when walking

PyPoE.poe.patchserver.node_check_hash(directory_node, folder_path=None, ggpk=None, recurse=True, bufsize=1048576)[source]¶

Compare expected hash against files on disk.

Folder hash is sha256sum of concaterated items in folder.

Accepts either a string folder path or a GGPKFile

Example GGPK:

from PyPoE.poe.file.ggpk import GGPKFile
ggpk_file = '/mnt/poe/Path_of_Exile/Content.ggpk'
ggpk = GGPKFile()
ggpk.read(ggpk_file)
ggpk.directory_build()

from PyPoE.poe.patchserver import node_check_hash
node_check_hash(ggpk['Data/Maps.dat'], ggpk=ggpk)

test_node = 'Art/2DArt/Atlas'

node = ggpk[test_node]

# GGPK files match GGPK hashes?
node_hashes = node_check_hash(node, ggpk=ggpk)

if node_hashes[-1][2] is True:
  print('hashes match for {}'.format(node.get_path()))
else:
  for child_node, child_hash, child_bool in node_hashes:
    if bool is False:
      print(child_node.record.name)
      print('{} file hash'.format(child_hash.hexdigest()))
      print('{:064x} expected hash'.format(child_node.record.hash))

# up to date?
import PyPoE.poe.patchserver
patch = PyPoE.poe.patchserver.Patch()
patch_file_list = PyPoE.poe.patchserver.PatchFileList(patch)
cdir = ''
for dir in test_node.split('/'):
  if len(cdir) > 0:
      cdir += '/'
  cdir += dir
  patch_file_list.update_filelist([cdir])

from hmac import compare_digest
node_patchserver_results = []
for index, child in enumerate(node_hashes):
    node_path = child[0].get_path()
    node_patchserver = patch_file_list.directory[node_path]
    node_file_hash = child[1]
    hash_test = compare_digest(
        node_file_hash.hexdigest(),
        format(node_patchserver.record.hash, '064x'))

    node_patchserver_results.append(hash_test)

if list(zip(node_hashes, node_patchserver_results))[-1][1] is True:
  print('hashes match patchserver for {}'.format(test_node))

Example files:

import os
import PyPoE.poe.patchserver

patch = PyPoE.poe.patchserver.Patch()
patch_file_list = PyPoE.poe.patchserver.PatchFileList(patch)
poe_dir = '/mnt/poe/Path_of_Exile'
node_hash_list = PyPoE.poe.patchserver.node_check_hash(
  patch_file_list.directory, folder_path=poe_dir, recurse=False)
for node, checksum, matched in node_hash_list:
  print('{} hashed okay?: {}'.format(
    node.record.name,
    matched))

Parameters:	directory_node (`PyPoE.poe.file.ggpk.DirectoryNode`) – The node to check that folder & files or GGPKFile contents match expected values. folder_path (str) – file system directory where files to check are located ggpk (`PyPoE.poe.file.ggpk.GGPKFile`) – GGPK file record recurse (bool) – If set, check folders bufsize (int) – The size of the buffer to use for computing each hash
Returns:	`PyPoE.poe.file.ggpk.DirectoryNode` : The tested node `hashlib` : sha256sum for file or folder `bool` : True if folder bytes hash == expected hash
Return type:	`list` [ `tuple` ]
Raises:	`ValueError` – One of either ggpk or folder_path must be given

PyPoE.poe.patchserver.node_outdated_files(patch_file_list, directory_node_path, folder_path, recurse=False, bufsize=1024)[source]¶

Get expected hash based list of outdated files on disk for given directory node.

Example:

import PyPoE.poe.patchserver
patch = PyPoE.poe.patchserver.Patch()
patch_file_list = PyPoE.poe.patchserver.PatchFileList(patch)
poe_dir = '/mnt/poe/Path_of_Exile'
node = ''
update_list = PyPoE.poe.patchserver.node_outdated_files(
    patch_file_list, node, poe_dir, recurse=False)
print(update_list)

Parameters:

patch_file_list (PatchFileList) – The connection to a patch server
directory_node_path (str) – The path name of the node PyPoE.poe.file.ggpk.DirectoryNode.get_path() to update folder & files for.
folder_path (str) – file system directory where files to check are located
recurse (bool) – If set, update all files in directories below directory_node_path
bufsize (int) – The size of the buffer to use for computing each hash

Returns:

hashlib:

list:: PyPoE.poe.file.ggpk.DirectoryNode

Return type:

dict

Internal API¶

PyPoE.poe.patchserver.socket_fd_open(socket_fd)[source]¶

Create a TCP/IP socket object from a socket.socket.detach() file descriptor. Uses socket.fromfd().

Parameters:	socket_fd (fd) – File descriptor to build socket from.
Returns:	socket
Return type:	`socket`

PyPoE.poe.patchserver.socket_fd_close(socket_fd)[source]¶

Shutdown (FIN) and close a TCP/IP socket object from a socket.socket.detach() file descriptor.

Parameters:	socket_fd (fd) – File descriptor for socket to close

class PyPoE.poe.patchserver.BaseRecordData(name, hash)[source]¶

Bases: PyPoE.shared.mixins.ReprMixin

Sibling to PyPoE.poe.file.ggpk.BaseRecord.

PyPoE.poe.file.ggpk.DirectoryNode.record item base class. Built from record data, rather than pointer details from GGPK file.

Used for each item detailed by patchserver.

Variables:	_name (str) – Name of item hash (int) – SHA256 hash of file contents

class PyPoE.poe.patchserver.VirtualDirectoryRecord(*args, **kwargs)[source]¶

Bases: PyPoE.poe.patchserver.BaseRecordData, PyPoE.poe.file.ggpk.DirectoryRecord

name¶

Returns and sets the name of the file

If setting, it also takes care of adjusting name_length accordingly. Returns name of the file.

Returns:	name of the file
Return type:	str

read(ggpkfile)¶

Read this record’s header for the given GGPKFile instance.

Parameters:	ggpkfile (GGPKFile) – GGPKFile instance

write(ggpkfile)¶

Write this record’s header for the given GGPKFile instance.

Parameters:	ggpkfile (GGPKFile) – GGPKFile instance

class PyPoE.poe.patchserver.VirtualFileRecord(name, hash, size)[source]¶

Bases: PyPoE.poe.patchserver.BaseRecordData, PyPoE.poe.file.ggpk.FileRecord

extract(buffer=None)¶

Extracts this file contents into a memory file object.

Parameters:	buffer (io.Bytes or None) – GGPKFile Buffer to use; if None, open the parent GGPKFile and use it as buffer.
Returns:	memory file buffer object
Return type:	io.BytesIO

extract_to(directory, name=None)¶

Extracts the file to the given directory.

Parameters:	directory (str) – the directory to extract the file to name (str or None) – the name of the file; if None use the file name as in the record.

name¶

Returns and sets the name of the file

If setting, it also takes care of adjusting name_length accordingly. Returns name of the file.

Returns:	name of the file
Return type:	str

read(ggpkfile)¶

Read this record’s header for the given GGPKFile instance.

Parameters:	ggpkfile (GGPKFile) – GGPKFile instance

write(ggpkfile)¶

Write this record’s header for the given GGPKFile instance.

Parameters:	ggpkfile (GGPKFile) – GGPKFile instance

Table of Contents

This Page

Overview¶

Description¶

Agreement¶

Documentation¶

Public API¶

Internal API¶