Skip to content

Last updated：2026-10-12

Protobuf ZeroCopy Serialization

1. ZeroCopyStream

protobuf has something called ZeroCopyStream at the IO interface layer. For IO interface design, protobuf provides related serialization and deserialization interfaces, such as:

// Read a protocol buffer from the given zero-copy input stream.  If
// successful, the entire input will be consumed.
bool ParseFromZeroCopyStream(io::ZeroCopyInputStream* input);

// Write the message to the given zero-copy output stream.  All required
// fields must be set.
bool SerializeToZeroCopyStream(io::ZeroCopyOutputStream* output) const;

The original design intent of ZeroCopyStream is to minimize the number of buffer copies, that is, to omit copying the data inside the stream into the user's buffer. Therefore, the stream can return a buffer that actually points directly to the final data structure where the bytes are to be stored, and the caller can interact with this buffer directly, eliminating the intermediate copy operation.

For example, the classic IO stream:

char buffer[[]BUFFER_SIZE];
input->Read(buffer, BUFFER_SIZE);
DoSomething(buffer, BUFFER_SIZE);

Then the stream basically just calls memcpy() to copy the data from protobuf into the user's buffer. If we use ZeroCopyInputStream, we only need:

const void* buffer;
int size;
input->Next(&buffer, &size);
DoSomething(buffer, size);

Here no copy is performed; the caller ends up reading directly from the buffer.

ZeroCopyStream provides two base classes: ZeroCopyOutputStream/ZeroCopyInputStream.

Taking InputStream as an example, we can usually customize our own ZeroCopyStream by inheritance, implementing the following four interfaces.

// implements ZeroCopyInputStream ----------------------------------
bool Next(const void** data, int* size);
void BackUp(int count);
bool Skip(int count);
int64 ByteCount() const;

Some rpc frameworks basically all customize their own Stream, for example: sofa-pbrpc

https://github.com/baidu/sofa-pbrpc/blob/master/src/sofa/pbrpc/buffer.h

2. Demo

Define a protobuf protocol, for example: authentication:

syntax = "proto3";

message BasicAuth {
  string username = 1;
  string password = 2;
}

Then write the serialization and deserialization:

  • Serialization
BasicAuth auth_message;
auth_message.set_username("user123");
auth_message.set_password("password");

StringOutputStream output_stream(&buf);

auth_message.SerializeToZeroCopyStream(&output_stream);
  • Deserialization
ArrayInputStream input_stream(buf.data(), buf.size());
BasicAuth auth_message;
if (!auth_message.ParseFromZeroCopyStream(&input_stream)) {
    std::cerr << "Failed to parse data." << std::endl;
}

std::cout << "Username: " << auth_message.username() << std::endl;
std::cout << "Password: " << auth_message.password() << std::endl;

Then we can call it:

int main() {
    std::string buf;
    buf.reserve(512);
    ser(buf);
    deser(buf);
    return 0;
}

Output:

Username: user123
Password: password

That's it for this section. See the Knowledge Planet community for the complete code above — welcome to discuss rpc-related topics with me.

(gdb) p *dsr
$8 = {typeInfo = {datumlen = 4, typid = 1082, align = 105 'i', byval = true}, datumStreamVersion = DatumStreamVersion_Original, nth = -1,
  logical_row_count = 16382, datump = 0x3840470 "\277\377\377\377\231\002", physical_datum_index = -1, physical_datum_count = 0,
  datum_beginp = 0x3840470 "\277\377\377\377\231\002", datum_afterp = 0x38503dc "", has_null = true, buffer_beginp = 0x383fc60 "",
  null_bitmap = {buffer = 0x383fc70 "", byteBit = 0 '\000', bytePointer = 0x383fc70 "", bitCount = 16382, bitPosition = -1,
    readBitOnCount = 58981488}, null_bitmap_beginp = 0x0, rle_can_have_compression = false, rle_compress_beginp = 0x0, rle_repeatcountsp = 0x0,
  delta_beginp = 0x0, delta_deltasp = 0x0, delta_datum_p = 0, rle_norepeats_null_bitmap_count = 0, rle_compress_bitmap_count = 0,
  rle_repeatcounts_count = 0, rle_repeatcounts_size = 0, delta_item = false, delta_bitmap_count = 0, deltas_count = 0, deltas_size = 0,
  rle_block_was_compressed = false, rle_compress_bitmap = {buffer = 0x0, byteBit = 0 '\000', bytePointer = 0x0, bitCount = 0, bitPosition = 0,
    readBitOnCount = 0}, rle_repeatcounts_index = 0, rle_in_repeated_item = false, rle_repeated_item_count = 0,
  rle_total_repeat_items_read = 0, delta_block_was_compressed = false, delta_bitmap = {buffer = 0x0, byteBit = 0 '\000',
    bytePointer = 0xff6c00454244 <error: Cannot access memory at address 0xff6c00454244>, bitCount = 0, bitPosition = 0,
    readBitOnCount = 13651600}, eyecatcher = "\000\000\000", physical_data_size = 0, maxDataBlockSize = 13652112,
  errdetailCallback = 0x35daeb0, errdetailArg = 0x34c4e18, errcontextCallback = 0x34c4e18, errcontextArg = 0x383fc60, memctxt = 0x1077c}

Comments